Skip to content

feat(autobahn): introduce EpochTrio for multi-epoch handling (CON-358)#3736

Open
wen-coding wants to merge 23 commits into
mainfrom
wen/autobahn_epoch_trio
Open

feat(autobahn): introduce EpochTrio for multi-epoch handling (CON-358)#3736
wen-coding wants to merge 23 commits into
mainfrom
wen/autobahn_epoch_trio

Conversation

@wen-coding

@wen-coding wen-coding commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Data flow

The system has three layers that each need epoch information, but for different reasons:

data and avail — each caches an EpochTrio{Prev, Current, Next}. CommitQCs and blocks arrive in a sliding window that straddles epoch boundaries: a QC at the last road of epoch N must verify against epoch N's committee, while the next one needs epoch N+1's. Both layers advance their cached trio atomically when a CommitQC crosses an epoch boundary.

consensus — caches a single *Epoch. Consensus processes one view at a time and always stays within a single epoch. It only needs one committee at a time; when a CommitQC crosses the boundary, it looks up the next epoch from the registry and replaces its cached pointer. Incoming votes are gated by the vote's own View().EpochIndex against this cached epoch — a vote for another epoch is dropped rather than verified against the wrong committee.

EVM shard routing — the p2p router reads data.EpochTrio() (returned by value) fresh on each send and uses Current.Committee().EvmShard(sender) to route incoming EVM transactions to the correct validator shard. Reading fresh means it always follows data's current epoch with no cached pointer to go stale.

Registry and boundary seedingAdvanceIfNeeded is driven by block execution: executeBlock (shared between validators and full nodes via giga_router_common.go) calls it after every executed block. A block executed in epoch N seeds epoch N+2 into the registry as a placeholder; epoch N+1 is seeded one epoch earlier by executing epoch N-1 blocks. Because execution is sequential, epochs are registered in increasing order without gaps.

At an epoch boundary, avail.PushCommitQC and data.PushQC resolve the next EpochTrio before advancing — a plain TrioAt fast path, falling back to Registry.WaitForEpoch(N+2) when a node's CommitQC stream has outrun its own block execution. Such a node stalls at the boundary (off any lock, so execution keeps seeding and wakes it) until N+2 is registered, rather than erroring. This makes the boundary correct for validators, full nodes, and catching-up nodes alike. Consensus itself only ever needs N+1 — always seeded before N+2 by sequential execution — so no consensus-side midpoint gate is required.

@cursor

cursor Bot commented Jul 10, 2026

Copy link
Copy Markdown

PR Summary

High Risk
Changes epoch-boundary behavior, per-epoch vote quorum, and blocking CommitQC ingest in avail/data when registry seeding lags execution—core finality and committee correctness paths.

Overview
Adds EpochTrio (Prev / Current / Next) so avail and data can verify and serve traffic across epoch boundaries without assuming a single committee.

Avail drops a fixed inner.epoch, keeps an atomic epochTrio, and reworks lane votes to accumulate per-epoch weight (current + next at ingest, applyEpoch back-fill when Next enters the window). Boundary CommitQCs resolve the next trio (with WaitForEpoch when execution lags), run advanceEpochLanes (add/remove lanes, retain Prev lanes for boundary QC collection), and wake waiters so lane QCs that formed in the incoming epoch are observed.

Data caches epochTrio, resolves epochs via EpochForRoad on PushQC / PushBlock, and advances the trio on the last road of an epoch with the same wait/retry pattern as avail.

Consensus restores the view epoch from the registry (separate commit vs view epoch at boundaries), rotates epoch on CommitQC, drops prepare/commit/timeout votes whose EpochIndex does not match the current epoch, and calls SealSeeding after startup. Persisted-inner validation now takes commit and view epochs separately.

Stale AppVote / out-of-window messages are dropped instead of failing peer streams where noted in the diff.

Reviewed by Cursor Bugbot for commit a3748e2. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedJul 14, 2026, 3:31 AM

@wen-coding wen-coding changed the title refactor(autobahn): replace trioAt() with direct epochTrio cache and remove LatestEpoch() (CON-358) autobahn epoch: introduce EpochTrio so we can handle multiple epochs (CON-358) Jul 10, 2026
Comment thread sei-tendermint/internal/autobahn/consensus/inner.go Outdated
Comment thread sei-tendermint/internal/autobahn/avail/state.go Outdated
Comment thread sei-tendermint/internal/autobahn/avail/state.go Outdated
Comment thread sei-tendermint/internal/autobahn/avail/state.go Outdated
@codecov

codecov Bot commented Jul 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 71.57623% with 110 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.97%. Comparing base (d9c188d) to head (a3748e2).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
sei-tendermint/internal/autobahn/avail/state.go 51.21% 29 Missing and 11 partials ⚠️
sei-tendermint/internal/autobahn/data/state.go 53.06% 11 Missing and 12 partials ⚠️
sei-tendermint/internal/autobahn/epoch/registry.go 74.13% 10 Missing and 5 partials ⚠️
...ei-tendermint/internal/autobahn/consensus/state.go 41.66% 10 Missing and 4 partials ⚠️
...ei-tendermint/internal/autobahn/consensus/inner.go 55.55% 5 Missing and 3 partials ⚠️
sei-tendermint/internal/autobahn/avail/inner.go 82.92% 3 Missing and 4 partials ⚠️
sei-tendermint/internal/p2p/giga_router_common.go 84.61% 1 Missing and 1 partial ⚠️
sei-tendermint/internal/autobahn/epoch/testonly.go 95.83% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3736      +/-   ##
==========================================
- Coverage   59.87%   58.97%   -0.91%     
==========================================
  Files        2286     2200      -86     
  Lines      189603   180001    -9602     
==========================================
- Hits       113523   106152    -7371     
+ Misses      65958    64512    -1446     
+ Partials    10122     9337     -785     
Flag Coverage Δ
sei-chain-pr 80.09% <71.57%> (?)
sei-db 70.41% <ø> (ø)
sei-db-state-db ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sei-tendermint/autobahn/types/epoch_trio.go 100.00% <100.00%> (ø)
...-tendermint/internal/autobahn/avail/block_votes.go 100.00% <100.00%> (+9.09%) ⬆️
...int/internal/autobahn/consensus/persisted_inner.go 97.59% <100.00%> (ø)
...ei-tendermint/internal/p2p/giga_router_fullnode.go 25.92% <100.00%> (-16.94%) ⬇️
...i-tendermint/internal/p2p/giga_router_validator.go 90.24% <100.00%> (-1.76%) ⬇️
sei-tendermint/internal/autobahn/epoch/testonly.go 96.77% <95.83%> (-3.23%) ⬇️
sei-tendermint/internal/p2p/giga_router_common.go 60.66% <84.61%> (+1.19%) ⬆️
sei-tendermint/internal/autobahn/avail/inner.go 92.66% <82.92%> (-4.88%) ⬇️
...ei-tendermint/internal/autobahn/consensus/inner.go 61.16% <55.55%> (-3.28%) ⬇️
...ei-tendermint/internal/autobahn/consensus/state.go 81.76% <41.66%> (-6.51%) ⬇️
... and 3 more

... and 90 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

seidroid[bot]
seidroid Bot previously requested changes Jul 10, 2026

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The EpochTrio refactor is broad and mechanically clean, but bounding epoch 0 to a finite road range now forces a real epoch transition at road 108,000, and the consensus trio-rotation has an off-by-one that will stall consensus at that boundary; there are also committee-staleness and nil-deref latent defects in the epoch-transition machinery.

Findings: 5 blocking | 4 non-blocking | 4 posted inline

Blockers

  • Cached placeholder committees survive real epoch registration (Codex P1). registry.TrioAt() generates genesis-committee placeholder epochs and inserts them; AddEpoch() later replaces only the map pointer, so EpochTrio values already cached in the avail/consensus/data inners keep pointing at the stale placeholder *Epoch. advanceEpoch() additionally early-returns whenever the road index is still inside Current's range, so it never refreshes a stale Next. Once a future epoch's committee actually differs from genesis, next-epoch vote weighting (avail reweightForNextEpoch / pushVote uses trio.Next.Committee()) and QC verification will use the wrong committee. Consider having AddEpoch mutate the existing epoch object in place, or make advanceEpoch/TrioAt re-fetch Next from the registry rather than trusting the cached pointer.
  • Cursor's second-opinion review (cursor-review.md) and the repo REVIEW_GUIDELINES.md are both empty, so no Cursor findings or repo-specific standards were available to incorporate; Codex's two findings were reviewed and are corroborated above.
  • Missing test coverage for the actual epoch-boundary transition: the new tests (TestInsertQCCrossEpochFallback, TestPushCommitQCCrossEpochFallback, TestAdvanceEpochTrio) only exercise the registry.TrioAt fallback with artificially-constructed trios, and every other test uses SingleEpochTrio. No test drives consensus across a real road-108,000 boundary (committing the last road of epoch e and proposing the first road of epoch e+1), which is exactly where the P0 off-by-one manifests.
  • 2 blocking issue(s) flagged inline on specific lines.

Non-blocking

  • Signature verification was moved under the hot inner lock in avail PushBlock and PushVote (previously done outside via VerifyInWindow). Holding s.inner across ed25519/BLS verification serializes all per-lane block/vote pushes. Prefer snapshotting the committee(s) under the lock and verifying outside, as PushCommitQC and data.PushQC already do.
  • avail.PushAppQC verifies against inner.epochTrio directly without the cross-epoch registry.TrioAt fallback that PushCommitQC uses, so an appQC/commitQC whose epoch is outside the cached trio would fail verification; confirm this cannot occur or add the same fallback for consistency.
  • 2 suggestion(s)/nit(s) flagged inline on specific lines.

iSend.Store(inner{persistedInner: persistedInner{CommitQC: utils.Some(qc)}, epoch: i.epoch})
// Reuse the cached trio when the QC stays within the current epoch.
trio := i.epochTrio
if !trio.Current.RoadRange().Has(qc.Proposal().Index()) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocker] Off-by-one at the epoch boundary (Codex P0). This condition keys on the committed QC's index, but the trio must cover the next view, which is qc.Proposal().Index()+1 (= View().Index). When committing the last road of epoch e (Index() == Last_e), Has(Last_e) is true, so the trio stays at epoch e; but View() then stamps the next view (road Last_e+1, belonging to epoch e+1) with epoch e's index. That proposal fails Proposal.Verify's road-range check on every node → consensus stalls. Since this PR bounds epoch 0 to [0, EpochLength-1] (was OpenRoadRange), the chain will hit this at road 108,000. Advance based on NextIndexOpt(qc) / the new View's index instead: if !trio.Current.RoadRange().Has(qc.Proposal().Index()+1).

return fmt.Errorf("unknown epoch_index %d", anchor.Proposal().EpochIndex())
var anchorLanes func(func(types.LaneID) bool)
for inner := range s.inner.Lock() {
ep, _ := inner.epochTrio.EpochFor(anchor.Proposal().EpochIndex())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocker] The ok from EpochFor is dropped and ep is dereferenced unconditionally. EpochFor returns (nil, false) when the epoch isn't in the trio, and Epoch.Committee() dereferences its receiver, so this nil-panics if the prune anchor's epoch has fallen outside the cached 3-epoch window. The previous code returned an explicit unknown epoch_index error here — restore an if !ok { return ... } guard.

return fmt.Errorf("block.Verify(): %w", err)
}
for inner, ctrl := range s.inner.Lock() {
if err := p.Msg().Verify(inner.epochTrio.Current.Committee()); err != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] PushBlock verifies the proposal only against epochTrio.Current.Committee(), whereas PushVote (below) accepts votes from either Current or Next. newInner explicitly initializes block queues for next-epoch-only lanes "so that early votes and proposals from validators joining in the next epoch are not rejected" — but a next-epoch-only proposer's block fails p.Msg().Verify(current) (HasLane) here and is rejected, contradicting that intent. Verify against Next as a fallback too, or drop the next-epoch block-queue init if only current-epoch proposals are intended.

if err := vote.Msg().Verify(currC); err != nil {
if err := vote.Msg().Verify(nextC); err != nil {
return fmt.Errorf("vote.Verify(): %w", err)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] VerifySig (and Msg().Verify above) now run while holding the inner lock. These crypto checks were previously performed outside the lock; on the hot vote path this serializes all pushes behind signature verification. Snapshot currC/nextC under the lock and verify after releasing it.

Comment thread sei-tendermint/internal/autobahn/consensus/inner.go Outdated

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional findings (outside current diff — PR may have been updated during review):

  • 🔴 sei-tendermint/internal/autobahn/avail/state.go:432-441 — PushBlock/PushVote regressed on two fronts vs. pre-PR: (1) signature verification (Ed25519/BLS) now runs inside s.inner.Lock(), serialising every peer block/vote against PushCommitQC, PushAppVote, ProduceLocalBlock, WaitForLaneQCs, RecvBatch, etc.; the pre-PR code intentionally routed this through registry.VerifyInWindow outside the lock. (2) PushBlock verifies only against inner.epochTrio.Current.Committee() — unlike sibling PushVote which does Current→Next fallback — so signed lane proposals from Next-only validators (whose queues newInner explicitly allocates) get rejected before enqueue. Fix by mirroring the pattern already used in PushCommitQC (state.go:277-287): snapshot the trio under a brief lock, run Verify/VerifySig outside the lock, then re-lock to enqueue.

    Extended reasoning...

    The regression\n\nBefore this PR, PushBlock and PushVote verified signatures via s.data.Registry().VerifyInWindow(...) outside s.inner.Lock(). After this PR (state.go:435-441 and 486-497), both p.Msg().Verify(committee) and the CPU-heavy p.VerifySig(committee) execute inside the for inner, ctrl := range s.inner.Lock() iterator body.\n\ns.inner is utils.Watch[*inner] (state.go:36). Watch.Lock() (libs/utils/mutex.go:214-220) acquires w.ctrl.mu.Lock() — a plain exclusive mutex, not an RWMutex — and holds it for the entire iterator body. Ed25519 verify is order-of-tens-of-microseconds per call; BLS is heavier. That work now serialises against roughly 26 other call sites (PushCommitQC, PushAppQC, PushAppVote, WaitForLaneQCs, ProduceBlock, RecvBatch, the persister loop, fullCommitQC, …) which all take the same mutex.\n\n### Concrete failure scenario\n\nA validator receives sustained peer traffic — say 100 blocks/s from lane producers and 3000 votes/s from replicas across lanes. Each verify is ~50µs of Ed25519 (worse for BLS). Under the new code the mutex is held for roughly (100+3000)×50µs ≈ 155 ms of wall-clock per second across PushBlock/PushVote alone, blocking every consensus wake-up (ctrl.Updated()) and every other s.inner writer during those windows. Tail latency on QC advancement (PushCommitQC), lane-QC construction (laneQC via RecvBatch), and app-vote acceptance (PushAppVote) all spike. PushVote is worse than PushBlock — it may run Verify+VerifySig twice (Current, then Next fallback) inside the same lock scope.\n\n### The correct pattern already lives in this file\n\nPushCommitQC (state.go:277-287) snapshots the trio under a brief lock, runs qc.Verify(trio) OUTSIDE the lock, and then re-enters to store. PushAppVote (state.go:308-319) uses the same pattern to pull the committee out before v.VerifySig. Committee and *Epoch values are immutable (only inner is atomically replaced), so snapshotting Current+Next committees before the verify is safe. Mirroring this pattern in PushBlock/PushVote is a few lines and closes the regression.\n\n### Secondary issue: PushBlock lacks the Next-committee fallback\n\nnewInner (inner.go:76-84) explicitly allocates blocks and votes queues for lanes present only in the Next epoch, with the comment "so that early votes and proposals from validators joining in the next epoch are not rejected." PushVote implements the Current→Next fallback matching that intent (state.go:486-497). PushBlock does not — it verifies only against Current (state.go:435-441). BlockHeader.Verify (block.go:75-80) fails on !c.HasLane(h.lane), so a signed lane proposal from a Next-only validator is rejected before it can be enqueued in the pre-allocated inner.blocks[lane] slot.\n\nStep-by-step: trio at road 0 with Current=ep0 committee {A,B,C,D} and Next=ep1 committee {B,C,D,E} (E is new). newInner allocates inner.blocks[E] and inner.votes[E] via the Next-only lane loop. E signs and pushes a LaneProposal for its lane. PushBlock reaches p.Msg().Verify(currC)BlockHeader.Verify!currC.HasLane(E) → returns "E is not a lane". inner.blocks[E] stays empty; peers whose PushVote fallback accepted votes for that would-be block now hold votes referencing a header that will never exist. When the transition completes and Current advances to ep1, E has to re-propose and any prior votes are useless. Note: Registry.AddEpoch is only invoked from tests today (production always fabricates Next as a placeholder sharing the genesis committee), so this doesn't manifest in production yet — but the plumbing this PR sets up is explicitly for real epoch transitions.\n\n### Suggested fix\n\nBoth issues fold into the same edit — mirror PushCommitQC:\n\ngo\nvar trio types.EpochTrio\nfor inner := range s.inner.Lock() {\n trio = inner.epochTrio\n}\ncurrC, nextC := trio.Current.Committee(), trio.Next.Committee()\nif err := p.Msg().Verify(currC); err != nil {\n if err := p.Msg().Verify(nextC); err != nil {\n return fmt.Errorf("block.Verify(): %w", err)\n }\n}\nif err := p.VerifySig(currC); err != nil {\n if err := p.VerifySig(nextC); err != nil {\n return fmt.Errorf("block.VerifySig(): %w", err)\n }\n}\nfor inner, ctrl := range s.inner.Lock() {\n // ... existing enqueue logic ...\n}\n\n\nSame shape for PushVote. This restores the pre-PR concurrency behaviour and gives PushBlock the Next-fallback that newInner was built to support.

Comment thread sei-tendermint/internal/autobahn/avail/state.go Outdated
Comment thread sei-tendermint/internal/autobahn/consensus/inner.go

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well-structured infrastructure PR that introduces the EpochTrio abstraction and fixes the live-path epoch advancement / restart-trio derivation; the security-critical QC verification correctly binds epoch_index ↔ committee ↔ road-range, and I found no reachable-today blockers. The two second-opinion findings are either about not-yet-reachable future work or contained by downstream checks, so they land as non-blocking notes.

Findings: 0 blocking | 4 non-blocking | 1 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • Cursor's second-opinion review file (cursor-review.md) is empty — that pass produced no output. Codex produced two findings, both incorporated below.
  • Codex (High): the EpochTrio caching design means a cached trio retains placeholder (genesis-committee) epochs generated by Registry.TrioAt, and reweightForNextEpoch/advanceEpoch never create new per-lane queues (blocks/votes/nextBlockToPersist/persistedBlockStart) for validators that only join in a later committee — so once real committee changes exist, new members' lanes would be permanently rejected with ErrBadLane. This is a genuine limitation but is NOT reachable in this PR: Registry.AddEpoch is defined but never called anywhere in production (confirmed by grep), so all epochs currently resolve to the genesis committee placeholder, and inner.go:14-18 already documents this exact gap as an explicit TODO. Recommend ensuring that follow-up (dynamic committee wiring) adds lane-queue creation on epoch transition and the avail runPersist cross-epoch-lane union (state.go:769-771 TODO) before AddEpoch is ever invoked in production.
  • avail.PushVote (state.go:497-531) verifies the LaneVote message and its signature against independently-chosen committees (message may validate against Current while the signature validates against Next), unlike PushBlock (state.go:443-453) which pins a single committee for both checks. Exploitability is limited today because currC==nextC in practice (no committee changes yet) and blockVotes.pushVote / inner.laneQC re-check membership via per-epoch weights, so no forged current-epoch LaneQC is possible; still worth tightening for consistency once committees can differ.
  • 1 suggestion(s)/nit(s) flagged inline on specific lines.

h := vote.Msg().Header()
for inner, ctrl := range s.inner.Lock() {
currC, nextC := inner.epochTrio.Current.Committee(), inner.epochTrio.Next.Committee()
if err := vote.Msg().Verify(currC); err != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] Consistency/robustness: unlike PushBlock (which pins a single committee for both the message and signature checks), PushVote validates vote.Msg().Verify and vote.VerifySig against independently-selected committees — the message can pass against currC while the signature only passes against nextC (or vice-versa), admitting a vote that is not valid under any single committee. It's not exploitable today (committees don't yet diverge, and blockVotes.pushVote/laneQC re-gate by per-epoch weight so no forged current-epoch QC results), but consider mirroring PushBlock: select one committee via Msg().Verify, then use that same committee for VerifySig.

@seidroid seidroid Bot dismissed their stale review July 10, 2026 03:37

Superseded: latest AI review found no blocking issues.

seidroid[bot]
seidroid Bot previously requested changes Jul 10, 2026

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A large, well-tested refactor replacing single-epoch *Epoch lookups with an EpochTrio/EpochTrioCursor abstraction across consensus/data/avail/p2p. The mechanics are sound for the current fixed-committee case, but the newly-added epoch-transition machinery has two latent correctness gaps (surfaced by Codex) that will bite once membership-changing transitions are actually wired up, plus a lock-scope regression in avail PushVote.

Findings: 1 blocking | 6 non-blocking | 2 posted inline

Blockers

  • None at the file/PR level.
  • 1 blocking issue(s) flagged inline on specific lines.

Non-blocking

  • Performance/concurrency regression in avail/state.go PushVote: signature verification (vote.Msg().Verify and vote.VerifySig) now runs while holding s.inner.Lock(). The previous code verified outside the lock via Registry().VerifyInWindow(...). PushBlock in the same file still verifies outside the lock and only takes the trio under the lock — PushVote should follow the same pattern to avoid serializing expensive signature checks under the state mutex.
  • cursor-review.md is empty — the Cursor second-opinion pass produced no output; only Codex's review was available to merge.
  • REVIEW_GUIDELINES.md (pulled from the base branch) is empty, so no repo-specific review standards could be applied.
  • Minor: several exported Verify methods lost their doc comments (BlockHeader.Verify, Block.Verify, LaneVote.Verify, LaneRange.Verify, View.Less, TimeoutVote.Verify, etc.). Not a lint failure under the configured linters, but the removals are incidental to the refactor and slightly reduce documentation.
  • registry.TrioAt inserts generated placeholder epochs into the shared map permanently (epochAt); benign given road indices come from verified QCs, but worth a note that far-future lookups would grow the map.
  • 1 suggestion(s)/nit(s) flagged inline on specific lines.

// TODO: filter votes per-epoch committee once epoch transitions are wired up.
// reweightForNextEpoch refreshes the per-epoch weight cache in every stored
// blockVotes entry for the given next epoch. Called once per epoch transition.
func (i *inner) reweightForNextEpoch(nextEpoch *types.Epoch) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocker] reweightForNextEpoch only recomputes weights for queues that already exist; it never creates votes/blocks queues for lanes first introduced by the new Next epoch. newInner seeds queues for the starting Current ∪ Next lanes, but after an Advance (here and in PushCommitQC), the trio's Next becomes a further epoch whose brand-new lanes have no queue. A valid vote/block from such a lane then passes committee verification against Next but fails the inner.votes[h.Lane()] / inner.blocks[...] lookup with ErrBadLane (avail/state.go PushVote/PushBlock). This is latent today (no production AddEpoch caller, and the transition test uses identical committees across epochs so no new lane appears), but it's a real defect in the transition path and should be fixed — create queues for newly-introduced next-epoch lanes on transition — before membership-changing epoch transitions are enabled. (Confirms Codex finding.)

}

// Trio returns the current cached trio.
func (c EpochTrioCursor) Trio() types.EpochTrio { return c.trio }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] EpochTrioCursor caches *Epoch pointers in trio. registry.TrioAt may return placeholder epochs built from the genesis committee when AddEpoch hasn't yet supplied the real one. Once such a placeholder is cached here, Trio() and the current-epoch fast path in ForVerify keep returning the genesis committee even after AddEpoch overwrites the registry entry — the cursor only refreshes on Advance into a different epoch. This surfaces in data.CurrentTrio()Committee().EvmShard(...) routing and in current-epoch QC/vote verification. Consider refreshing the cached trio from the registry (or not caching placeholder-backed epochs) so a later AddEpoch is observed. Latent until AddEpoch is wired to production, but worth addressing with the transition work. (Confirms Codex finding.)

Comment thread sei-tendermint/internal/autobahn/data/state.go Outdated
Comment thread sei-tendermint/internal/autobahn/consensus/inner.go Outdated

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well-structured refactor replacing single-Epoch verification with an EpochTrio/EpochTrioCursor model across the autobahn layers, with good new test coverage. No shipping-blocking bugs: the multi-epoch concerns raised are currently latent because AddEpoch/non-genesis committees have no production callers yet, so all epochs resolve to the genesis-committee placeholder. A few consistency and locking notes are worth addressing.

Findings: 0 blocking | 5 non-blocking | 2 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • Cursor second-opinion pass produced no output (cursor-review.md is empty); REVIEW_GUIDELINES.md is also empty, so no repo-specific guidelines were applied.
  • Codex (High) — cached trios retain overridden placeholder epochs: Registry.epochAt inserts placeholders using the genesis committee, and cursors cache the resulting EpochTrio. Once AddEpoch overwrites a registry entry with the real committee, existing cursors keep pointing at the stale placeholder (Next in particular) until they Advance. This is currently latent (no production AddEpoch caller and all committees are the genesis placeholder), but it will produce wrong committees for next-epoch vote weighting / QC verification once real epoch transitions are wired up. Consider a way for cursors/TrioAt to pick up committee updates, or document/enforce the ordering constraint.
  • Registry.TrioAt mutates the map (inserts placeholders) as a side effect of a read-style lookup; over many distinct road indices this grows the map with placeholder epochs that are never GC'd. Minor, but worth a note since it also underlies the staleness issue above.
  • 2 suggestion(s)/nit(s) flagged inline on specific lines.

inner.qcs[inner.nextQC] = qc
inner.nextQC += 1
}
inner.cursor.Advance(qc.QC().Proposal().Index())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] Codex (Medium): data advances the cursor with qc.QC().Proposal().Index(), whereas consensus (pushCommitQC) and avail (PushCommitQC) advance with Index()+1. After finalizing the last QC of an epoch, insertQC (line 226) and this path leave the cursor on the old epoch until the first QC of the next epoch arrives, so CurrentTrio() — consumed by the p2p EVM shard router (giga_router_common.go/giga_router_validator.go) — reports the previous committee across the boundary window. Verification itself still works via the ForVerify Next-range fallback, and it's harmless today (all committees are the genesis placeholder), but the divergence from the other two layers looks unintentional; either align on Index()+1 or add a comment explaining why data intentionally lags.

}
committee = nextC
}
if err := vote.VerifySig(committee); err != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] PushVote now runs both structural Verify and the cryptographic VerifySig inside the s.inner.Lock() scope, so signature verification serializes on the state mutex. Note PushBlock (just above) deliberately reads the committees under the lock, releases it, and verifies outside. Consider matching that pattern here to keep expensive crypto off the hot lock.

@seidroid seidroid Bot dismissed their stale review July 10, 2026 04:13

Superseded: latest AI review found no blocking issues.

Comment thread sei-tendermint/internal/autobahn/avail/inner.go
Comment thread sei-tendermint/internal/autobahn/avail/state.go Outdated
…-358)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@wen-coding wen-coding force-pushed the wen/autobahn_epoch_trio branch from c36ec91 to d03b4f5 Compare July 13, 2026 03:42
@wen-coding wen-coding changed the title autobahn epoch: introduce EpochTrio so we can handle multiple epochs (CON-358) feat(autobahn): introduce EpochTrio for multi-epoch verification (CON-358) Jul 13, 2026
@wen-coding wen-coding changed the title feat(autobahn): introduce EpochTrio for multi-epoch verification (CON-358) feat(autobahn): introduce EpochTrio for multi-epoch handling (CON-358) Jul 13, 2026
Comment thread sei-tendermint/internal/autobahn/data/state.go Outdated

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR introduces EpochTrio to move autobahn from a single-epoch to a three-epoch (Prev/Current/Next) view, replacing LatestEpoch()/EpochByIndex with road-index-based window lookups and adding epoch seeding/advancement. The core mechanics look sound and are well tested, but two edge cases flagged by Codex are worth addressing: reweightForNextEpoch under-initializes lane queues, and PushQC now rejects out-of-window QCs; both are latent (masked while all epochs share the genesis committee) but will matter once real committee rotation lands.

Findings: 0 blocking | 6 non-blocking | 2 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • Cursor produced no output: cursor-review.md is empty. REVIEW_GUIDELINES.md is also empty, so no repo-specific standards were applied. (Codex did produce output — both its findings are incorporated below.)
  • The PR description references APIs/tests that are not present in the diff — advanceEpoch(), data.State.CurrentTrio(), TestInsertQCCrossEpochFallback, TestPushCommitQCCrossEpochFallback, TestAdvanceEpochTrio, and a "cross-epoch verification fallback" in data.PushQC. The actual code exposes EpochTrio() and advances via TrioAt(idx+1) with no fallback. The description appears stale relative to the committed changes; update it so reviewers/maintainers aren't misled about what shipped.
  • Nit: several error messages wrap EpochForRoad failures with the label EpochAt(%d) (e.g. avail/state.go PushCommitQC/PushAppVote/PushAppQC). Rename for accurate diagnostics.
  • Nit: TestPushAppQCPreviousEpoch (avail/state_test.go) builds epochN but discards it (_ = epochN) and never actually advances the state into epoch N, so it does not exercise the "late AppQC after an epoch boundary crossed" scenario its comment describes — it only pushes an epoch N-1 QC/AppQC into a genesis-seeded state.
  • 2 suggestion(s)/nit(s) flagged inline on specific lines.

Comment thread sei-tendermint/internal/autobahn/avail/inner.go Outdated
Comment thread sei-tendermint/internal/autobahn/data/state.go
wen-coding and others added 2 commits July 12, 2026 20:49
…xt lanes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR wires multi-epoch handling via a new EpochTrio abstraction and a seeding-aware Registry; the core logic and test coverage are solid. No present-tense blockers (all epochs still use the genesis committee placeholder), but there are two latent correctness gaps that will bite once real committee rotation is enabled, plus minor notes.

Findings: 0 blocking | 6 non-blocking | 2 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • Cursor second-opinion review (cursor-review.md) and the repo REVIEW_GUIDELINES.md were both empty — no Cursor output to merge and no repo-specific guidelines applied.
  • Codex flag (confirmed, latent): fullnodes never seal the registry. SealSeeding() is only called from consensus.NewState (validator path); NewGigaFullnodeRouter builds only a data.State and no consensus/avail state, so on a fullnode seeding stays true forever and EpochAt/TrioAt will auto-generate any missing epoch with the genesis committee, permanently bypassing AdvanceIfNeeded's advancement constraints. Harmless today (genesis committee everywhere) but silently defeats this PR's own safety gate. Consider sealing the registry on the fullnode path (or documenting why it must remain open).
  • After SealSeeding, Registry.EpochAt still acquires the write lock (state.Lock()) for pure reads because it may auto-generate during seeding; consensus hot paths (pushProposal midpoint gate, pushCommitQC) call it per-message. Minor serialization/perf concern worth a fast read path once seeding is sealed.
  • Several wrapped errors are mislabeled: e.g. state.go PushCommitQC/PushAppVote/PushAppQC return fmt.Errorf("EpochAt(%d): ...") while actually calling EpochForRoad. Cosmetic, but confusing in logs.
  • 2 suggestion(s)/nit(s) flagged inline on specific lines.

Comment thread sei-tendermint/internal/p2p/giga_router_fullnode.go Outdated
Comment thread sei-tendermint/internal/autobahn/avail/inner.go
wen-coding and others added 3 commits July 12, 2026 21:02
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…xtEpoch

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Comment thread sei-tendermint/internal/autobahn/consensus/inner.go Outdated
Comment thread sei-tendermint/internal/autobahn/data/state.go
…h split

Track lane votes per epoch instead of reweighting the current-epoch set at
each boundary. pushVote credits a vote to the Current and Next epochs it is
valid under (via a shared credit helper); advanceEpochLanes back-fills a
newly-entering epoch's sets from stored votes so a block finalized under it
(lagging lane, or shared committee) still reaches quorum. laneQC reads the
current epoch's set directly. pushVote reports quorum only for the current
epoch; the boundary CommitQC's ctrl.Updated() surfaces the incoming epoch's
already-formed laneQCs. headers() recovers block headers from votes[0].

consensus/newInner: verify the persisted CommitQC against its own epoch while
stamping the current view (and its votes/QCs) with the next-view epoch, so a
CommitQC on the last road of an epoch validates on restart.

data.State.EpochTrio returns by value; giga routers read it fresh instead of
caching, and share a newGigaRouterCommon constructor. Align avail EpochForRoad
error labels; map a lost-epoch race in fullCommitQC to ErrPruned.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment thread sei-tendermint/internal/autobahn/avail/inner.go
WaitForLaneQCs iterates the proposal epoch's lanes but passed the cached trio
to laneQC, which read vote weight and quorum under trio.Current. When the
proposal epoch and trio.Current differ (across an epoch boundary), that reads
the wrong per-epoch vote set. laneQC now takes the epoch directly so committee,
quorum, epoch index, and lane set are all resolved from the same epoch.

Also document that byKey is retained as the applyEpoch back-fill source and
freed only when the block number is pruned.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean, well-tested wiring of the new EpochTrio abstraction into the avail and consensus layers, replacing single-epoch assumptions with road-index window lookups. No correctness blockers found; a few minor, non-blocking observations.

Findings: 0 blocking | 4 non-blocking | 0 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • Cursor's second-opinion review (cursor-review.md) produced no output; Codex reported no material issues. This synthesis is based on my own static analysis plus those passes.
  • avail/state.go NewState restart path: the epoch-boundary catch-up calls advanceEpochLanes(nextTrio) exactly once and only back-fills nextTrio.Next. This is correct today because votes are not persisted (vote queues are empty on restart, so applyEpoch is a no-op), but it would become subtly wrong if vote persistence is ever added and a restart's loaded QCs span more than one epoch boundary. Worth a comment or a guard so the assumption is explicit.
  • EpochTrio.all()/CurrentAndNextLanes assume Current and Next are non-nil (guaranteed by Registry.TrioAt). A nil Current/Next would panic in ep.RoadRange()/ep.Committee() rather than erroring; the invariant is documented but not defensively enforced. Low priority given all production trios come from TrioAt.
  • The midpoint liveness gate in consensus/inner.go is explicitly best-effort (a node catching up via CommitQC sync past the midpoint bypasses it). This is documented and acceptable, but it means the N+2-seeded guarantee relies on the execution/AdvanceIfNeeded path rather than the gate itself — flagging for awareness.

Comment thread sei-tendermint/internal/autobahn/avail/state_test.go
… erroring

In-order processing (waitForCommitQC bounds idx to at most the Next epoch) means
a road outside the Prev/Current/Next window can only be below it — a QC already
committed. EpochForRoad failing there is not an error; return nil to skip it,
matching the stale idx != commitQCs.next case below.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment thread sei-tendermint/internal/autobahn/avail/state.go
Comment thread sei-tendermint/internal/autobahn/avail/state.go

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A large, well-tested refactor that introduces EpochTrio for multi-epoch handling across data/avail/consensus and a seeding Registry. The core mechanics are sound and heavily unit-tested; the main concerns are two latent robustness/correctness edges in avail/state.go (a persist-goroutine termination under epoch lag, and a TOCTOU between vote verification and per-epoch crediting) that are benign under today's placeholder identical committees but should be hardened before real committee rotation lands.

Findings: 0 blocking | 6 non-blocking | 2 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • REVIEW_GUIDELINES.md and cursor-review.md are both empty/missing, so no repo-specific standards and no Cursor second-opinion pass were available. Codex's two findings (below) were reviewed and confirmed as real.
  • Several EpochForRoad lookups on the live epochTrio window hard-fail their goroutine when a road falls outside the Prev/Current/Next window (e.g. Run at state.go:~722, runPersist at state.go:813). Since the Registry never deletes epochs, these anchor/QC epoch lookups could use s.data.Registry().EpochAt(idx) instead, which stays valid for older epochs and avoids window-lag fragility. Worth making the pattern consistent across these paths.
  • Test coverage gap: the new avail epoch-boundary path in PushCommitQC (advanceEpochLanes + the 'only wake' ctrl.Updated() that surfaces lane QCs already at quorum in the incoming Current epoch) has no integration test that actually crosses an epoch boundary with lane QCs forming in the new epoch. The delicate 'do not make it conditional or the boundary stalls' comment suggests this deserves a regression test.
  • The midpoint liveness gate (consensus/inner.go) triggers only on exact equality with RoadRange().MidPoint(); the code comments correctly note catch-up-via-sync nodes bypass it. This is acceptable but relies on the seeding invariant (N+2 seeded by executeBlock) holding; a defensive log/metric when the gate actually fires would aid observability.
  • 2 suggestion(s)/nit(s) flagged inline on specific lines.

ep, epOK := s.data.Registry().EpochByIndex(anchor.Proposal().EpochIndex())
if !epOK {
return fmt.Errorf("unknown epoch_index %d", anchor.Proposal().EpochIndex())
ep, err := s.epochTrio.Load().EpochForRoad(anchor.Proposal().Index())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] Robustness (confirmed, from Codex): resolving the batch anchor's epoch through the live s.epochTrio.Load() window makes runPersist fragile. This is a scope.SpawnNamed("persist", ...) goroutine (state.go:707), so returning an error here permanently terminates persistence (and tears down the avail component). If persistence ever falls more than two epochs behind consensus, EpochForRoad(anchorRoad) returns an error even though the anchor was valid when collected — a durability/liveness hazard rather than a transient skip. Since the Registry never deletes epochs, prefer s.data.Registry().EpochAt(anchor.Proposal().Index()) here (only the anchor's lane set is needed), which stays resolvable regardless of how far the persist loop lags.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AppQC and CommitQC should never be more than one epoch apart, so we should be fine here.

// in the window under which its signer has weight. Passing Current and
// Next means a next-epoch validator's vote is counted for the next epoch
// immediately, so no reweight is needed when the epoch advances.
trio := s.epochTrio.Load()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] TOCTOU (confirmed, from Codex): the vote is verified against the trio loaded outside the lock at line 518, but credited against a freshly reloaded trio here inside the lock. PushCommitQC advances s.epochTrio under the same inner lock, so a boundary advance between the two loads causes the vote to be credited to {Current=N+1, Next=N+2} after being validated only against {N, N+1} — i.e. credited (and potentially counted toward a LaneQC) for epoch N+2, which VerifyInWindow never checked. This is benign today because all epochs share the genesis committee (credit()'s per-epoch Weight check makes it a no-op or a still-valid vote), but it becomes a real correctness bug once committees actually rotate. Capture the trio once before verifying and reuse that same snapshot for crediting (or re-validate after acquiring the lock).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is safe even though verify and credit can see different trio snapshots. credit re-checks Weight(signer) per epoch, so weight only accrues to an epoch where the signer is actually a member — and the signature that VerifyInWindow already proved is epoch-independent (it's over the block header, checked against the signer's pubkey, which is the same identity in any committee it belongs to). So crediting against an advanced {N+1, N+2}:

if the signer has no weight there, credit adds nothing — the vote just stays in byKey;
if it does have weight there, the already-proven signature is valid there too, so the credit is legitimate.
Even a set credited for a lane not in the new epoch is never read, since laneQC is only queried for lanes in the epoch's committee. And the reverse (the vote missing from epoch N's set) is harmless because the trio only advances past N once CommitQC(N.Last) is committed — every epoch-N laneQC that was needed has already formed, so nothing is still waiting on one.

So it can't admit an invalid vote or drop a needed one. Capturing a single snapshot would also strand the vote from the concurrent applyEpoch back-fill, so I've left the two loads as-is and will add a comment explaining the invariant.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional findings (outside current diff — PR may have been updated during review):

  • 🟡 sei-tendermint/internal/autobahn/avail/state.go:339-343 — PushAppVote resolves the vote's epoch via EpochForRoad before checking whether the AppVote is stale, so a redundant/duplicate AppVote whose road has fallen more than one epoch behind can hard-error instead of being silently dropped.

    Extended reasoning...

    PushAppVote (sei-tendermint/internal/autobahn/avail/state.go:339-343) calls s.epochTrio.Load().EpochForRoad(idx) at the very top of the function, using the vote's road index. EpochForRoad only searches the bounded Prev/Current/Next window (roughly 3 epochs, ~36h at current epoch length). The "already superseded" staleness check — idx < types.NextOpt(inner.latestAppQC), which is the actual signal that this vote carries no new information — only runs later, inside the lock, after EpochForRoad, VerifySig, and waitForCommitQC have all already executed.

    The consequence is that a stale/redundant AppVote whose road has fallen outside the 3-epoch window returns a hard fmt.Errorf("EpochForRoad(%d): %w", ...) instead of being dropped. That error propagates out of PushAppVote and up through clientStreamAppVotes (sei-tendermint/internal/p2p/giga/avail.go), tearing down the entire RunClient connection to that peer (all its streams — consensus, lane votes, commitQCs, blocks — not just AppVotes), rather than just skipping one useless vote.

    This is a genuine regression versus the pre-PR code, which resolved the epoch via registry.EpochByIndex(v.Msg().Proposal().EpochIndex()) — a lookup against the full registry map, where old epochs are never evicted. Under the old code an old vote's epoch always resolved successfully, and the vote was then dropped gracefully by the staleness check further down. The new windowed EpochForRoad lookup turns that graceful drop into a hard failure once the vote's road falls more than one epoch behind Current.

    The file already establishes the correct pattern for handling exactly this situation: fullCommitQC (same file, around line 600) wraps an EpochForRoad miss as data.ErrPruned with a comment explaining that the epoch window can advance past a QC's road, and that should be treated as "pruned" rather than an error. PushAppVote does not follow this pattern — it treats an identical window-miss as a hard error instead of a benign, ignorable case.

    Step-by-step proof: Consider a validator A that is caught up, and a peer B whose execution frontier is lagging (e.g. recovering from a stall). B's live AppVote broadcast (serverStreamAppVotes via SubscribeAppVotes) streams votes from B's own execution frontier, which may be far behind A's. When A's clientStreamAppVotes goroutine receives one of B's AppVotes for a road that has fallen 2+ epochs behind A's current epochTrio.Current, PushAppVote calls EpochForRoad(idx) first — before ever checking idx < NextOpt(inner.latestAppQC) — and returns a hard error, even though A already has a full AppQC for that road and the vote is entirely superseded. The error kills A's entire RunClient connection to B, including consensus, block, and QC streams that would otherwise help B catch up faster.

    Fix: move the road-index staleness comparison against the latestAppQC watermark ahead of the EpochForRoad call (so a vote that is already superseded is dropped without ever needing epoch resolution), or — mirroring fullCommitQC — treat an EpochForRoad miss in PushAppVote as data.ErrPruned and drop the vote silently rather than propagating a hard error.

Comment thread sei-tendermint/internal/autobahn/avail/state.go
Comment thread sei-tendermint/internal/autobahn/avail/state.go
Comment thread sei-tendermint/internal/autobahn/consensus/state.go
Move byHash entry (and per-epoch set) creation into credit, so an entry exists
only once a weighted vote is actually appended. A vote with no weight in any
credited epoch — reachable when a verify/credit trio mismatch at a boundary
credits against a committee the signer left — now stays only in byKey and
creates no byHash entry. headers() therefore never observes an empty per-epoch
map. Add a laneVoteSet.add unit test and a zero-weight pushVote test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
seidroid[bot]
seidroid Bot previously requested changes Jul 14, 2026

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR wires real multi-epoch handling into the autobahn availability/consensus layers via a new EpochTrio abstraction and per-epoch vote accumulation; the code is well-structured and thoroughly unit-tested, but there is one significant liveness risk (shared with Codex): the epoch-boundary advance in avail fails fatally if epoch N+2 is not yet seeded, and the only guard (the midpoint gate) is bypassable on the CommitQC-sync / full-node path.

Findings: 3 blocking | 4 non-blocking | 2 posted inline

Blockers

  • Epoch-boundary advance can fatally crash a catching-up node. avail.State.PushCommitQC calls TrioAt(idx+1) (needs N+2) at the boundary and returns the error on failure; that error propagates up through clientStreamCommitQCs/storeCommitQC (giga/avail.go:200, consensus/state.go:300) and terminates the scope. N+2 is only seeded by block execution (AdvanceIfNeeded in executeBlock), which is decoupled from — and can lag behind — the CommitQC stream that advances inner.commitQCs.next. The consensus midpoint gate (consensus/inner.go:227) only protects actively-voting validators and is explicitly acknowledged to be bypassed on the CommitQC-sync path and absent on full nodes. If the CommitQC stream reaches N.Last before execution executes any block in epoch N, the boundary advance errors and the node halts. Either gate every boundary advance on N+2 presence (skip/wait rather than error), or seed N+2 independently of execution progress. This matches Codex's High finding.
  • 2 blocking issue(s) flagged inline on specific lines.

Non-blocking

  • cursor-review.md was empty (Cursor produced no output); Codex produced a single High finding, which is incorporated above.
  • The midpoint gate keys on exact road-index equality (Index() == MidPoint()). If the proposal at exactly the midpoint road is never processed (view timeout / skipped), the gate never fires even for a validator. Consider gating on >= MidPoint() (fire once past the midpoint) so the check is not dependent on hitting one exact road.
  • NewState defaults startRoadIdx to 0 with a TODO noting production nodes should restart from a snapshot; genesis-only correctness is fine for now but worth tracking as a follow-up before mainnet.
  • The N+2 back-fill in advanceEpochLanes re-runs applyEpoch over every block number of every retained lane queue on each boundary crossing (O(lanes×blocks×votes)); boundaries are rare (every EpochLength roads) so this is acceptable, but worth a comment noting the cost is intentionally bounded to boundary events.

// catches up past the midpoint via CommitQC sync (without processing the
// midpoint proposal) bypasses the check; that is acceptable because such a
// node has already executed enough blocks to seed N+2 anyway.
if proposal.Proposal().Msg().Index() == i.epoch.RoadRange().MidPoint() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocker] This midpoint gate is the only mechanism ensuring N+2 is seeded before a boundary advance, but as the comment above admits it is best-effort: it only fires for a validator that processes the exact midpoint proposal. Full nodes (no voting) and any node catching up via CommitQC sync bypass it entirely. Combined with the fatal error path in avail.State.PushCommitQC (TrioAt(idx+1) → returned error terminates the CommitQC stream scope), a node whose execution lags the CommitQC stream past an epoch boundary will crash rather than stall. The assumption that a catch-up node "has already executed enough blocks to seed N+2 anyway" is not guaranteed — AdvanceIfNeeded (execution) is decoupled from commitQCs.next (stream-driven). Recommend gating every boundary advance (not just the voting path) or seeding N+2 independently of execution.

if idx == s.epochTrio.Load().Current.RoadRange().Last {
nextTrio, err := s.data.Registry().TrioAt(idx + 1)
if err != nil {
return fmt.Errorf("TrioAt(%d): %w", idx+1, err)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocker] Returning an error here is fatal: PushCommitQC errors bubble up to clientStreamCommitQCs (giga/avail.go:200) and the storeCommitQC iterator (consensus/state.go:300), tearing down the scope. Since commitQCs.next can reach the boundary purely from the CommitQC stream while block execution (which seeds N+2 via AdvanceIfNeeded) lags, this "fail loudly" branch can halt a catching-up or full node. Prefer waiting/skipping until N+2 is present over a hard error, or ensure N+2 seeding is ordered before the stream can deliver the boundary QC.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beyond the inline finding above, this run also examined three other candidate issues in avail and data and ruled them out: unprotected/racy EpochForRoad lookups that could tear down the avail component on a 2-epoch race, runPersist treating an EpochForRoad failure as fatal (unlike its sibling fullCommitQC), and PushQC's justifying comment for erroring on a stale EpochForRoad lookup appearing to contradict the documented cursor invariant. None of these were confirmed as real bugs.

Extended reasoning...

This is a continuation of an already extensively reviewed, safety-critical consensus PR (epoch-boundary handling across autobahn's data/avail/consensus layers) with a long history of findings and several still-open maintainer threads. This run surfaced one new inline finding — a stale-epoch verification race in consensus.pushCommitQC that directly rebuts the maintainer's earlier 'CommitQCs arrive strictly sequentially' dismissal by tracing that the feed is actually populated by avail's independent, batching persist goroutine (and the p2p stream on validators), not synchronously by consensus's own QC generation. Beyond that finding, three other candidate issues around EpochForRoad usage in avail/state.go and data/state.go were examined this run and ruled out as not real bugs. Given the severity and volume of prior findings in this PR and the unresolved design threads with pompon0, this is not a PR to shadow-approve; recording the ruled-out items here purely for the record so they are not re-explored from scratch.

Comment on lines 156 to 168
if qc.Proposal().Index() < i.View().Index {
return nil
}
// CommitQC advances to new index; clear all state for new view.
// TODO: rotate ep when epoch transitions are wired up.
iSend.Store(inner{persistedInner: persistedInner{CommitQC: utils.Some(qc)}, epoch: i.epoch})
nextEp, err := i.registry.EpochAt(qc.Proposal().Index() + 1)
if err != nil {
logger.Error("next epoch not in registry at CommitQC boundary",
"road", qc.Proposal().Index()+1)
return fmt.Errorf("EpochAt(%d): %w", qc.Proposal().Index()+1, err)
}
iSend.Store(inner{persistedInner: persistedInner{CommitQC: utils.Some(qc)}, registry: i.registry, epoch: nextEp})
}
return nil
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 consensus.pushCommitQC (inner.go:151) verifies an incoming CommitQC against the cached i.epoch before resolving the QC's own epoch from its road index, unlike avail.PushCommitQC and data.PushQC which both already do the road-index resolution first. Because the feed (avail.LastCommitQC().Iter) is a coalescing watch fed in part by an independent batching persist goroutine (and, on validators, by the p2p network stream), a CommitQC that jumps past an epoch boundary can be observed here and deterministically fail Verify, tearing down the whole consensus scope (avail, propose, outputs, storeCommitQC, etc.).

Extended reasoning...

The bug. consensus.pushCommitQC (sei-tendermint/internal/autobahn/consensus/inner.go:146-164) does:

func (s *State) pushCommitQC(qc *types.CommitQC) error {
	i := s.innerRecv.Load()
	if qc.Proposal().Index() < i.View().Index {
		return nil
	}
	if err := qc.Verify(i.epoch); err != nil {          // <- verifies against the STALE cached epoch
		return fmt.Errorf("qc.Verify(): %w", err)
	}
	for iSend := range s.inner.Lock() {
		i := iSend.Load()
		...
		nextEp, err := i.registry.EpochAt(qc.Proposal().Index() + 1)  // <- only resolves the epoch for the NEXT stored inner
		...
		iSend.Store(inner{... epoch: nextEp})
	}
	return nil
}

qc.Verify(i.epoch) runs before any road-index-based epoch resolution. The registry.EpochAt(qc.Proposal().Index()+1) call a few lines later only decides which epoch to store for the next round — it never corrects the verification that already happened. i.epoch is advanced one QC at a time (EpochAt(prevIdx+1)), which is only correct if pushCommitQC observes every CommitQC in strict sequence with no gaps.

Why that assumption is false. pushCommitQC is driven by s.avail.LastCommitQC().Iter (consensus/state.go:309), which is an AtomicRecv/coalescing watch (libs/utils/mutex.go, atomicWatch.Iter): each iteration does a fresh Load() of the current value; any Store() calls that land while the previous iteration's callback is still running are overwritten and never individually replayed. The next Load() simply returns whatever is latest.

avail.State's latestCommitQC is Stored exclusively from markCommitQCsPersisted (avail/state.go:871), which is the afterEach callback of CommitQCPersister.MaybePruneAndPersist, invoked back-to-back for every QC in a persist batch inside the runPersist goroutine — i.e. it is not driven synchronously by consensus's own QC generation, and it can fire many times in a tight loop with no yield between calls. On a validator, avail.PushCommitQC also accepts CommitQCs from the p2p network stream (clientStreamCommitQCs), not just from local generation.

Why the maintainer's dismissal doesn't hold. The author replied twice that this is a false positive, on the premise that LastCommitQC is populated directly by consensus's own storeCommitQC goroutine with strictly sequential indices, so no boundary can be skipped. Tracing the code shows that premise is incorrect: latestCommitQC.Store is called from the independent, batching persist goroutine (and can also be fed by the network stream), not synchronously from consensus's local QC-generation loop. Queue-gaplessness (the underlying commitQCs queue is filled one entry at a time) is a different property from watch-observability (the Iter reader can still skip over intermediate Stores) — the dismissal conflates the two.

Concrete failure. During catch-up (e.g. a validator reconnecting after being offline, ingesting a backlog of persisted CommitQCs spanning an epoch boundary), several Store()s can land while pushCommitQC's callback for an older QC is still executing. The next Iter iteration then observes a QC whose road index has already crossed one or more epoch boundaries that i.epoch was never advanced through. qc.Verify(i.epoch) calls Proposal.Verify/View.Verify, which strictly requires the QC's EpochIndex to match i.epoch.EpochIndex() and its road to fall in i.epoch's range — both fail deterministically for the skipped-ahead QC.

Step-by-step proof. Assume EpochLength = 108,000 roads, ep0 = [0, 107999], ep1 = [108000, 215999]. (1) Consensus's cached i.epoch = ep0 (it last processed a QC at road ≤107998, so i.epoch = EpochAt(prevIdx+1) still resolves to ep0). (2) During catch-up, avail's persist goroutine runs markCommitQCsPersisted back-to-back for QCs at roads 107999, 108000, 108001, ... in the same tight loop, while consensus's Iter callback for an earlier QC is still executing (e.g. blocked briefly on the inner lock or scheduler contention). (3) The next Iter iteration loads the latest stored value — say the QC at road 108001 (epoch 1) — skipping straight past the boundary QC at 107999 that would have advanced i.epoch to ep1. (4) pushCommitQC calls qc.Verify(i.epoch=ep0); View.Verify sees got=1 (from the QC), want=0 (ep0.EpochIndex()) and returns an error. (5) That error is returned from the Iter callback, which propagates out of scope.SpawnNamed("pushCommitQC", ...) in consensus.State.Run, and scope.Run cancels every sibling task (avail, propose, outputs, storeCommitQC, pushPrepareQC, pushTimeoutQC) — the whole consensus component goes down.

Impact. This hits exactly the liveness-critical catch-up/epoch-boundary path the mechanism exists to serve, and is not gated behind committee rotation — the road-range/EpochIndex check in View.Verify fails purely from the road-index mismatch, independent of committee membership. It's timing-dependent (needs the coalescing to actually skip the boundary QC), so not guaranteed on every crossing, but plausible under load/GC pauses/persist-batch timing, and deterministic once it happens.

Fix. Resolve the QC's own epoch from its road index before verifying — e.g. ep, err := i.registry.EpochAt(qc.Proposal().Index()) and qc.Verify(ep) — mirroring the pattern avail.PushCommitQC and data.PushQC already use (both resolve via road index / EpochForRoad before calling Verify). Consensus is currently the sole outlier that verifies against a purely sequentially-advanced cached pointer instead of a road-derived lookup.

Add Registry.WaitForEpoch, backed by a highestEpoch AtomicSend bumped in
makeEpoch, so a node whose CommitQC stream outran its own block execution can
wait for the next epoch's committee to be seeded instead of hard-erroring.

PushCommitQC resolves the next trio before taking the inner lock: a plain
TrioAt fast path (almost always already seeded), falling back to WaitForEpoch
+ retry only on a miss. The wait holds no lock, so execution keeps seeding and
wakes it; only CommitQC ingest pauses meanwhile.

This makes the boundary correct for full nodes and catching-up nodes, not just
actively-voting ones, so the consensus midpoint gate is no longer needed and is
removed. Also graceful-drop stale PushAppVote/PushCommitQC out-of-window roads
with an INFO log, and return the resolved epoch from fullCommitQC to drop a
redundant lookup in the Run pusher.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A well-tested refactor introducing EpochTrio for multi-epoch handling; no confirmed blocking bug, but data.PushQC advances nextQC before a fallible TrioAt without the WaitForEpoch fallback its sibling avail.PushCommitQC uses (a robustness gap on the full-node sync path), and the added RoadRange.MidPoint is dead code that contradicts the PR description.

Findings: 0 blocking | 5 non-blocking | 2 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • Cursor's second-opinion review (cursor-review.md) is empty — that pass produced no output, so only the Codex second opinion was available to merge.
  • data.PushBlock now resolves the committee via epochTrio.EpochForRoad(storedQC.road) instead of registry.EpochByIndex. If a block is delivered during deep catch-up whose road has fallen more than one epoch behind Current (e.g. a retried/slow GetBlock) yet is still >= inner.first, EpochForRoad errors and giga/data.go tears down the block-sync connection, whereas the old EpochByIndex lookup succeeded. Low confidence — worth confirming the prune cursor never retains blocks older than the Prev epoch.
  • The diff is large (+1650/-441) but overwhelmingly mechanical test churn from the GenRegistry signature change; the substantive logic changes are comparatively small, well-commented, and well-covered by new tests. No prompt-injection or suspicious instructions found in the diff or PR description.
  • 2 suggestion(s)/nit(s) flagged inline on specific lines.

}
idx := qc.QC().Proposal().Index()
if idx == s.epochTrio.Load().Current.RoadRange().Last {
nextTrio, err := s.cfg.Registry.TrioAt(idx + 1)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] This boundary advance mutates inner.nextQC (lines 431-434) before calling the fallible TrioAt(idx+1), and unlike the sibling avail.PushCommitQC (which resolves nextTrio with a WaitForEpoch fallback before taking the inner lock) there is no wait here. If TrioAt fails (epoch N+2 not yet seeded), PushQC returns an error with nextQC already advanced past the boundary. On the full-node sync path giga/data.go:clientStreamFullCommitQCs has no avail gate and tears down the connection; on retry/reconnect the boundary QC now has needQC == false, so this if needQC block never runs again and epochTrio is stuck at Current=N — ingestion then fails permanently once roads reach N+2 (matching Codex P1).

In the common case this is masked by the blocksCacheSize (4000) backpressure in the needQC closure, which keeps execution within 4000 blocks of ingestion so N+2 is normally seeded in time. But the asymmetry with the avail path is a latent landmine in a consensus-critical path. Recommend mirroring avail: resolve nextTrio (with a WaitForEpoch(Current.EpochIndex()+2) fallback) before mutating nextQC, then advance nextQC and store the trio atomically so a failure can never leave partially-mutated, non-recoverable state.

Comment thread sei-tendermint/autobahn/types/epoch.go Outdated
func (r RoadRange) Has(idx RoadIndex) bool { return idx >= r.First && idx <= r.Last }

// MidPoint returns the road index at the midpoint of the range.
func (r RoadRange) MidPoint() RoadIndex { return r.First + (r.Last-r.First)/2 }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] RoadRange.MidPoint is dead code — it is not referenced anywhere in the tree (not even in tests). The PR description states consensus "at road MidPoint(N) refuses to sign any proposal unless epoch N+2 is present," but that gate is not implemented; consensus/inner.go:pushProposal instead documents that no gate is needed because avail.PushCommitQC blocks on WaitForEpoch. Either wire up the midpoint gate as described or drop this method and correct the PR description so the two don't diverge (Codex P2).

@seidroid seidroid Bot dismissed their stale review July 14, 2026 02:24

Superseded: latest AI review found no blocking issues.

… data

Consensus vote handlers (PushPrepareVote/CommitVote/TimeoutVote) now resolve the
committee from the authoritative innerRecv epoch and compare the vote's own
View().EpochIndex against it. A vote for a non-current epoch is dropped (Debug
log, since peers may stream arbitrary-epoch votes) instead of being verified
against the wrong committee, which would fail and tear down the peer's consensus
stream. Removes the async-myView-lag window across an epoch boundary.

data.PushQC mirrors avail.PushCommitQC: resolve the next trio (TrioAt fast path,
WaitForEpoch fallback) before taking the lock and before mutating nextQC, so a
lookup failure can never strand nextQC past an unfinished boundary.

Remove RoadRange.MidPoint, dead since the midpoint gate was dropped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 6f27ad2. Configure here.

Comment thread sei-tendermint/internal/autobahn/avail/state.go

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Large, carefully-documented and well-tested refactor that introduces EpochTrio (Prev/Current/Next) so consensus, data, and avail can verify QCs/votes/blocks across epoch boundaries and back-pressure on epoch seeding. No hard blockers found; the main items are reconciling the PR description's "midpoint gate" with the boundary-based back-pressure actually implemented, and confirming a couple of concurrency invariants. (I could not compile/run tests in this sandbox, so compilation was verified by inspection only.)

Findings: 0 blocking | 7 non-blocking | 2 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • PR-description vs implementation mismatch (Codex High): the description says consensus refuses to sign at MidPoint(N) unless epoch N+2 is seeded, but pushProposal implements no such gate and the added comment states none is needed because avail.PushCommitQC/data.PushQC block on WaitForEpoch at the boundary instead. This appears to be an intentional design change (and is safe today since all epochs use the genesis committee placeholder), but the description should be updated so the stated safety property matches the code. Confirm that stalling at the boundary rather than the midpoint is the intended back-pressure point.
  • Cursor produced no output (cursor-review.md is empty); only Codex's second-opinion review was available to merge.
  • testonly.go in package epoch has no build tag, so GenRegistry, GenRegistryAt, and LatestEpoch are compiled into production binaries even though they are documented as test-only. Pre-existing pattern, but consider a //go:build test-style guard or moving them to a _test.go/export_test helper to keep test-only surface out of prod.
  • registryState.latest is now only ever set by test helpers (makeRegistryAt) and read by the test-only LatestEpoch; production code paths (makeEpoch/AdvanceIfNeeded/EpochAt) never advance it. It is effectively dead state in production — worth a comment or removal to avoid confusion.
  • During the seeding phase EpochAt/TrioAt will auto-generate arbitrarily-high epoch indices on demand (e.g. EpochAt(5*EpochLength)), all with the genesis committee. This is intended for cold start, but once real committee rotation lands (the noted TODO) this must be revisited so a stray high road index cannot silently fabricate a wrong-committee epoch.
  • 2 suggestion(s)/nit(s) flagged inline on specific lines.

Comments that couldn't be anchored to the diff

  • sei-tendermint/internal/autobahn/data/state.go:1117 -- [suggestion] Unlike the analogous stale-QC handling in avail.PushCommitQC (which logs and returns nil for a road that has fallen out of the window), PushQC returns the EpochForRoad error directly. The comment argues a below-window QC would be a no-op, but returning an error here propagates up the s.data.PushQC task and can terminate the node on a merely-stale QC. Consider treating an out-of-window road as a skip (return nil) for consistency with the avail path, unless in-order delivery truly guarantees this never fires.

if i.View() != proposal.View() || i.TimeoutVote.IsPresent() || i.PrepareVote.IsPresent() {
return nil
}
// No epoch-seeding gate is needed here: avail.PushCommitQC blocks

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] Codex flagged this as the missing "midpoint seeding gate." The comment here says no gate is needed because avail.PushCommitQC/data.PushQC block on WaitForEpoch at the boundary. That reasoning holds for the N+2 dependency (avail/data need Next=N+2, gated by WaitForEpoch), and consensus only needs N+1 here (seeded during N-1 execution, always present), so pushCommitQC's EpochAt(Index+1) won't error at the boundary. The concern is that the PR description still advertises an explicit midpoint refuse-to-sign gate that no longer exists. Please reconcile the description with this implementation so the documented safety invariant matches reality.

// in the window under which its signer has weight. Passing Current and
// Next means a next-epoch validator's vote is counted for the next epoch
// immediately, so no reweight is needed when the epoch advances.
trio := s.epochTrio.Load()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Codex noted that VerifyInWindow (line 554, off-lock) and this pushVote credit (in-lock) load epochTrio separately, so a concurrent boundary advance can credit a vote against a trio different from the one that verified it. This looks safe as written: signatures are committee-independent (verified once via vote.VerifySig), credit re-gates each epoch on Committee.Weight(vote.Key()) != 0, and epochTrio only advances forward — so a vote can only be credited to an epoch it is genuinely a member of, and a future-only signer is rejected by VerifyInWindow before ever reaching here. Worth a brief comment documenting that the credit intentionally re-derives weight per epoch (rather than trusting the verify-time trio) so this invariant isn't lost in a future refactor.

…ants

Expand WaitForEpoch's doc to explain why waiting on the single highestEpoch
value is sufficient: sequential block execution registers epochs without gaps
(N+1 is seeded before N+2), so highestEpoch >= target implies target is present.

Document at the PushVote call site that credit intentionally re-derives weight
per epoch, so the off-lock verify / in-lock credit trio mismatch is safe, with a
note not to refactor it to trust the verify-time trio.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment thread sei-tendermint/internal/autobahn/avail/inner.go

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A large, carefully-implemented refactor introducing EpochTrio for multi-epoch handling across the autobahn data/avail/consensus layers and the epoch Registry, with extensive test coverage. No blocking correctness issues found; the main note is that the PR description documents a MidPoint epoch-seeding gate that the code intentionally omits in favor of boundary back-pressure, plus a consensus/data asymmetry in how missing epochs are handled.

Findings: 0 blocking | 6 non-blocking | 2 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • PR-description vs. implementation mismatch (raised by Codex as P1): the description says consensus refuses to sign proposals at MidPoint(N) unless epoch N+2 is seeded, but pushProposal explicitly omits any such gate and instead relies on back-pressure — data.PushQC / avail.PushCommitQC block on WaitForEpoch at the epoch boundary. The code path is self-consistent and the boundary wait is argued to be bounded by execution, so this reads as a stale description rather than a code bug. Recommend updating the PR/description (and the design doc) to match the boundary-back-pressure design that actually shipped. (Cursor and Codex reviews: Codex flagged only this; Cursor produced no output.)
  • The seeding argument that N+1 is always present when consensus commits the last road of epoch N holds only while execution lags consensus by less than ~2 epochs. It rests on AdvanceIfNeeded being driven off b.FinalAppState in executeBlock; worth confirming FinalAppState is present on every executed block so seeding cannot silently stall.
  • REVIEW_GUIDELINES.md (base branch) and cursor-review.md are both empty — no repo-specific review standards were available to apply and the Cursor second-opinion pass produced no findings. Focused Go tests could not be run here (toolchain download is network-blocked), so test outcomes were reviewed statically only.
  • GenRegistry now starts at a random epoch index (0 or 1); this is a nice fuzzing touch, but tests that still build QC chains from road 0 rely on epoch 0 always being in-window and on the placeholder committee being identical across epochs. If real per-epoch committees are later introduced, these tests will need to anchor their chains explicitly.
  • 2 suggestion(s)/nit(s) flagged inline on specific lines.

if i.View() != proposal.View() || i.TimeoutVote.IsPresent() || i.PrepareVote.IsPresent() {
return nil
}
// No epoch-seeding gate is needed here: avail.PushCommitQC blocks

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] This comment documents that no midpoint epoch-seeding gate exists here, which contradicts the PR description ("at road MidPoint(N), it refuses to sign any proposal unless epoch N+2 is already present"). The shipped design instead applies back-pressure at the epoch boundary via WaitForEpoch in data.PushQC/avail.PushCommitQC. The implementation is internally consistent — flagging only so the PR description / design doc gets reconciled with the code, since a future reader trusting the description would look for a gate that isn't here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR description has been updated.

// CommitQC advances to new index; clear all state for new view.
// TODO: rotate ep when epoch transitions are wired up.
iSend.Store(inner{persistedInner: persistedInner{CommitQC: utils.Some(qc)}, epoch: i.epoch})
nextEp, err := i.registry.EpochAt(qc.Proposal().Index() + 1)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Asymmetry worth a note: at the epoch boundary this hard-errors (tearing down consensus) if epoch N+1 isn't seeded, whereas data.PushQC and avail.PushCommitQC gracefully block on WaitForEpoch until seeding catches up. Under normal lag N+1 is guaranteed seeded (it's seeded while executing epoch N-1 blocks), so this is effectively defense-in-depth — but if execution ever falls ~2 epochs behind, consensus fails hard rather than stalling. Consider either waiting here for symmetry, or adding a comment stating the invariant that makes this unreachable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should fail hard if execution falls >1 epoch behind, because if committee is unknown we will make the wrong decisions.

markBlockPersisted runs as an async callback after disk I/O; if an epoch advance
deletes the lane in between, writing nextBlockToPersist[lane] would recreate a
bare, orphaned map entry the cleanup loop never reclaims. Skip when the lane is
absent, mirroring PushBlock/PushVote's lane gating.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
seidroid[bot]
seidroid Bot previously requested changes Jul 14, 2026

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid, well-tested refactor introducing EpochTrio for multi-epoch handling across data/avail/consensus/registry. One correctness blocker: data.PushQC can regress epochTrio via an unconditional store gated on a stale needQC (asymmetric with avail.PushCommitQC's guard). Cursor's second-opinion review produced no output.

Findings: 1 blocking | 3 non-blocking | 1 posted inline

Blockers

  • None at the file/PR level.
  • 1 blocking issue(s) flagged inline on specific lines.

Non-blocking

  • Consensus vote-drop re-delivery (consensus/state.go PushPrepareVote/PushCommitVote/PushTimeoutVote): the comment claims a dropped vote is "re-delivered once this node is in that epoch," but sendUpdates (giga/consensus.go) only re-sends when the sender's watched value changes or the connection resets — a stable connection will not re-send the same dropped vote. In practice a boundary-lagged node recovers via view-timeout re-voting (or reconnect), so this is a minor one-view liveness hiccup at epoch boundaries (~every 108k roads), NOT the permanent quorum failure Codex's High rates it. Recommend softening the comment to describe timeout-based recovery rather than automatic re-delivery.
  • Cursor's second-opinion review file (cursor-review.md) was empty — that pass produced no output. Codex flagged two Highs: its data/state.go point is confirmed (the blocker), and its consensus/state.go point is captured above at reduced severity.
  • EpochLength (108_000) and the genesis epoch's now-bounded RoadRange have become load-bearing constants (replacing the former OpenRoadRange). This is a placeholder pending real committee rotation / snapshot-based seeding (several TODOs acknowledge this); acceptable for this PR but worth tracking so the placeholder length matches production before epoch rotation ships.

inner.nextQC += 1
}
if nextTrio != nil {
s.epochTrio.Store(nextTrio)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocker] epochTrio can regress here. This s.epochTrio.Store(nextTrio) is gated only on needQC, which is computed under the first lock (line 403) and never re-checked under this second lock. avail.PushCommitQC guards the equivalent store with if idx != inner.commitQCs.next { return nil }, but PushQC has no such re-check — the for inner.nextQC < gr.Next loop correctly no-ops for an already-applied QC, yet this Store still fires.

Concurrent callers exist: on a full node multiple peer clientStreamFullCommitQCs goroutines (giga/data.go:39) call PushQC concurrently. Interleaving: caller A and B both read needQC=true for boundary QC at road N.Last and both resolve nextTrio=trio(N+1); A stalls (e.g. blocked in WaitForEpoch) after releasing the first lock; B advances nextQC and, as its stream proceeds across the next boundary, advances epochTrio to trio(N+2); A then wakes, re-enters this block with stale needQC=true, the for-loop no-ops, but this line stores trio(N+1) — regressing epochTrio from N+2 to N+1. That corrupts EvmProxy shard routing (EpochTrio().Current) and subsequent EpochForRoad-based QC/block verification.

Gate the store on this call actually performing the advance (e.g. only when inner.nextQC == gr.First at lock time), or reject a lower-epoch trio (monotonic guard), mirroring avail's idx != inner.commitQCs.next early return.

needQC is computed under an earlier lock, so a concurrent PushQC caller (multiple
peer streams) could re-enter the boundary block with a stale needQC and store an
older trio after another caller already advanced epochTrio, corrupting EvmProxy
routing and EpochForRoad verification. Re-check under the insert lock that this
call actually applies the QC range (nextQC == gr.First) before storing the trio,
mirroring avail's commitQCs.next guard.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A large, well-tested refactor introducing EpochTrio for multi-epoch handling across the autobahn data/avail/consensus layers; the design consistently resolves epochs from each QC's own road index and handles boundary seeding via WaitForEpoch. No confirmed blocker, but one defensive-hardening suggestion (raised by Codex) and a couple of process notes.

Findings: 0 blocking | 3 non-blocking | 0 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • REVIEW_GUIDELINES.md is empty in this checkout, so no repo-specific standards were available to apply; cursor-review.md is also empty (no Cursor second-opinion output). Codex produced one finding, addressed inline.
  • Most of the +1723/-444 diff is mechanical test churn from the GenRegistry signature change (now returns a third startEpoch value) and registry.LatestEpoch()registry.EpochAt(0)/TrioAt(...) migrations. These look correct and consistent; no action needed.
  • NewRegistry now bounds the genesis epoch to [0, EpochLength-1] instead of OpenRoadRange(). This relies on SealSeeding + AdvanceIfNeeded seeding future epochs; any road lookup for an unseeded epoch after sealing now returns an error that callers must handle (most do so by dropping stale QCs/votes). Worth confirming there is no remaining caller that assumed the old open-ended genesis range covers arbitrary roads.

Comments that couldn't be anchored to the diff

  • sei-tendermint/internal/autobahn/consensus/inner.go:151 -- [suggestion] pushCommitQC verifies qc against the cached i.epoch (the current view's epoch) rather than the epoch of the QC's own road index. This is the one place in the PR that still verifies a QC against a cached epoch pointer instead of resolving it from the QC — everywhere else now uses EpochForRoad/EpochAt(qc.Index()) precisely to avoid stale-epoch mismatches. If consensus ever observes a CommitQC that is more than one index ahead across an epoch boundary (Codex flags coalescing of the avail.LastCommitQC latest-value watch, and the restart path), i.epoch would be the prior epoch and qc.Verify fails on both the epoch_index binding (proposal.go:109) and the now-bounded RoadRange check (proposal.go:112), returning an error that terminates the consensus goroutine.

I could not construct a concrete reproduction: view advancement is serialized (each CommitQC must pass through pushCommitQC before the next can form, and the QC at road N.Last is still epoch N), which bounds the normal gap to +1 and keeps it in-epoch. So this is likely not reachable today — but the fix is trivial and removes the fragility: verify against i.registry.EpochAt(qc.Proposal().Index()) (the value already computed one line below for the next epoch), matching the pattern the rest of this PR adopts.

@seidroid seidroid Bot dismissed their stale review July 14, 2026 03:18

Superseded: latest AI review found no blocking issues.

…ToPersist

nextBlockToPersist is populated lazily by markBlockPersisted itself, so gating
its update on the entry pre-existing made every lane's first write a no-op —
nextBlockToPersist never advanced, collectPersistBatch re-collected from 0, and
the block persister panicked ("out of sequence"). Gate on blocks[lane] instead
(created per-lane in newInner, deleted only on decommission), mirroring
PushBlock's lane check.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A large, well-structured, and heavily-tested change introducing EpochTrio to enable multi-epoch handling across the autobahn epoch/data/avail/consensus layers. No blocking correctness issues found; the one second-opinion finding is a latent invariant rather than a live bug.

Findings: 0 blocking | 4 non-blocking | 0 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • Codex finding (registry.go WaitForEpoch): WaitForEpoch(N) gates only on the monotonic highestEpoch counter and does not verify epoch N is actually present in the map, while AdvanceIfNeeded seeds epochs with a +2 stride (executing an epoch-N block seeds N+2). In isolation AdvanceIfNeeded(0) sets highestEpoch=2 while epoch 1 is absent, so WaitForEpoch(1) would return successfully for a missing epoch. In production this is not reachable: epoch N+1 is seeded during init (TrioAt at NewState) and by sequential execution before N+2, and the boundary path re-checks membership via TrioAt, which errors (rather than silently proceeding) on a premature wake. Worth a defensive assertion or a comment tightening the invariant, since the contiguity guarantee is implicit and undefended.
  • The cursor-review.md and REVIEW_GUIDELINES.md files were empty, so no Cursor second-opinion pass or repo-specific guidelines were available to merge; review proceeded on Claude + Codex findings only.
  • Several NewState/PushQC paths now return hard errors from EpochForRoad/TrioAt where the old code returned an 'unknown epoch_index' error too; behavior is preserved and these are internal (local pusher) rather than peer-stream paths, so tear-down risk is low — but consider whether a road that has legitimately fallen just below the window in data.State.PushQC should be treated like the ErrPruned/stale-drop cases used elsewhere (avail.PushAppVote/PushCommitQC) for consistency.
  • Crash-recovery epochTrio reconstruction is asymmetric between layers (data uses lastQC.Index()+1 as initRoad; avail uses the raw anchor index then conditionally advances via commitQCs.next). Both appear to converge on the same trio, but this is subtle and multi-boundary restart cases are hard to fully verify by inspection — worth a dedicated restart-across-boundary integration test if not already covered.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is a large, consensus-critical change (multi-epoch handling for autobahn's avail/data/consensus layers), and given its scope and history I don't think it's appropriate for a bot to shadow-approve, even though this run's bug hunt found nothing new.

What was reviewed this round: the epoch-boundary trio advance/rollback paths in avail.PushCommitQC and data.PushQC, the consensus vote-epoch gating in PushPrepareVote/CommitVote/TimeoutVote, and the lane-queue lifecycle in advanceEpochLanes/markBlockPersisted. Several previously-flagged high-severity issues (epochTrio regression under concurrent PushQC callers, WaitForEpoch N+1 vs N+2 seeding, stale myView committee at boundaries) appear to have already been fixed in the most recent commits on this branch.

A few architectural questions raised by pompon0 (AtomicSend/Recv vs atomic.Pointer for epochTrio, blocking vs error semantics on epoch-not-yet-seeded) were explicitly deferred to an offline 1:1 rather than resolved in-thread, and this is exactly the kind of design call that should stay with human reviewers.

Extended reasoning...

Overview

This PR introduces EpochTrio (Prev/Current/Next) so the avail, data, and consensus autobahn layers can operate correctly across epoch boundaries, plus a Registry seeding/sealing mechanism for epoch bootstrap. It touches core BFT consensus plumbing: CommitQC/AppQC verification, lane vote quorum accounting, WAL restart/recovery, and p2p EVM shard routing — 29 files, with the highest-churn files being avail/state.go, avail/inner.go, consensus/inner.go, data/state.go, and epoch/registry.go.

Security risks

No injection/auth-bypass/data-exposure risk in the traditional sense, but the risk profile here is liveness and safety of the consensus protocol itself: an incorrect epoch-boundary transition can stall block production (deadlock at an epoch boundary) or, in the worst case, allow verification against the wrong committee. The PR's own review thread surfaced and fixed roughly a dozen such issues (nil-panics on unresolved epochs, verify-before-fallback ordering bugs, restart-time epoch regression, off-by-one road indices at boundaries) over multiple rounds — a strong signal of how easy this class of bug is to introduce here.

Level of scrutiny

This warrants the highest level of scrutiny this repo has: it is core consensus/finality logic, not a peripheral feature, and several of the fixed bugs were "deterministic chain-liveness failure at the first epoch boundary" class issues that unit tests didn't catch (committee-rotation-dependent bugs are latent today because all epochs share the genesis committee). Given that, and per this review's own guidance to never approve consensus-critical, complex changes, this is squarely in defer-to-human territory regardless of what a single bug-hunting pass finds.

Other factors

The timeline shows extensive, iterative review from Cursor Bugbot, a "seidroid" reviewer, and prior runs of this same claude review system, with the human maintainer (wen-coding) fixing issues commit-by-commit up through the current HEAD (a3748e2). Cross-checking git history, the most recent unresolved blocker in the thread (seidroid's "epochTrio can regress here" on concurrent PushQC callers) does appear fixed by a later commit (35a28b1), and the cursor/claude findings about WaitForEpoch N+1 vs N+2 seeding and stale myView committee usage also appear addressed by 6f27ad2. However, a handful of design questions from pompon0 (a core reviewer) — e.g. whether epochTrio should use AtomicSend/AtomicRecv instead of atomic.Pointer, and whether epoch-lookup misses should block vs error — were explicitly punted to an offline 1:1 rather than resolved in the PR thread itself. That, combined with the sheer size/criticality of the change, is reason enough to keep a human in the loop here rather than have this system shadow-approve.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants