Follower sync deadlock: failed multi-height fetch skips a height, consensus wedges permanently (v0.7.3)

[arjun215-eng]

Summary

A follower node (CL in --follow mode) can deadlock permanently when a batched sync fetch fails at the wrong moment. On a failed fetch, sync's sync_height is advanced past all heights in the failed range, but only the first height of the range is re-requested. If the range covered more than one height, the later height is never requested again. Consensus then waits forever at that height while sync fills all max_parallel_requests=5 slots with later blocks that can never be delivered, so no new requests are ever issued. The services stay active — the node just silently stops advancing.

Observed on v0.7.3 (built from source), Arc testnet, chain id 5042002. The node wedged at height 52893874 and sat stalled for ~18 hours until the CL was manually restarted (restart fully recovers it).

Environment

• arc-node v0.7.3, built from the release tag via cargo install
• CL: arc-node-consensus start ... --follow --follow.endpoint https://rpc.drpc.testnet.arc.network,... --execution-persistence-backpressure
• EL: arc-node-execution node --chain arc-testnet --disable-discovery ... (IPC to CL)
• Ubuntu 22.04.5, EL/CL on the same host

Timeline (2026-07-21, UTC)

The node had been sync-following normally. Certificate not found (-32004) fetch failures are chronic in our logs and are normally harmless because the failed height gets re-requested — until this sequence:

05:08:00.949949  sync{tip=52893872 sync_height=52893873}: Requesting sync from peer range=52893873..=52893873 peer=1Ak9...
05:08:00.957722  WARN Fetch failed request_id=39161 error="Failed to fetch certificates: JSON-RPC error: code -32004: Certificate not found"
05:08:00.957760  sync{tip=52893872 sync_height=52893874}: Re-requesting values from peer request_id=39161 except=1Ak9...
05:08:01.185749  sync{tip=52893872 sync_height=52893873}: Requesting sync from peer range=52893873..=52893874 peer=1Ak9...   <-- 2-height range
05:08:01.195753  WARN Fetch failed request_id=39162 error="... code -32004: Certificate not found"
05:08:01.195778  sync{tip=52893872 sync_height=52893875}: Re-requesting values from peer request_id=39162 except=1Ak9...     <-- sync_height jumped PAST 52893874
05:08:01.195788  sync{tip=52893872 sync_height=52893875}: Requesting sync from peer range=52893873..=52893873 peer=1AWR...   <-- only 52893873 re-requested
05:08:01.321114  ✅ Fetch completed, sending blocks to consensus request_id=39163 start_height=52893873 count=1
05:08:01.348720  consensus{height=52893873}: Decided round=0 height=52893873
05:08:01.362328  consensus{height=52893874}: Starting new height height=52893874
05:08:01.689196  sync{tip=52893873 sync_height=52893875}: Requesting sync from peer range=52893875..=52893875 ...            <-- 52893874 never requested again

The failed request 39162 covered 52893873..=52893874. Its failure bumped sync_height to 52893875, but the retry (39163) only covered 52893873..=52893873. Height 52893874 fell through the gap.

Sync then fetched 52893875–52893879 into its 5 parallel slots. Consensus can't decide 52893874 (no value, and as a follower with role=None it can't propose), those 5 pending deliveries never complete, and sync refuses to issue new requests:

05:08:04.364143  consensus{height=52893874 round=0}: Timeout elapsed step=Propose
05:08:06.364464  consensus{height=52893874 round=0}: Timeout elapsed step=Rebroadcast    (repeats forever)
...
22:52:21.033505  sync{tip_height=52893873 sync_height=52893880}: SYNC REQUIRED: Falling behind tip_height=52893873 sync_height=52893880 peer_height=52987354
22:52:21.033517  sync{...}: Maximum number of parallel requests reached, skipping request for values max_parallel_requests=5 pending_requests=5

Meanwhile the EL just waits (its view of the incident):

22:47:11  WARN Beacon client online, but no consensus updates received for a while. ... period=63549.767870051s

Expected behavior

Either (a) a failed range fetch re-requests the entire failed range, not just its first height, or (b) sync detects tip_height + 1 < min(pending heights) and re-requests the gap, or (c) pending requests time out and free their slots so the gap can eventually be re-fetched. Any of these would prevent the permanent wedge.

Possibly related

During the same session (but ~2h earlier, and recovered automatically via the serial fallback) the EL hit a panic in the parallel state root task on v0.7.3:

thread 'tokio-rt' (1593146) panicked at .../reth-e231042ee7db3fb7/d6324d6/crates/trie/sparse/src/parallel.rs:3049:36:
node at path Nibbles(0x71) does not exist
02:50:17.736866  WARN Failed to compute state root in parallel

Happy to provide fuller logs for either issue.

[osr21]

Confirmed on Arc Testnet (client-side observations) + fix recommendation + mitigation

We've been running a keeper service against Arc Testnet v0.7.3 for several weeks making frequent eth_call / eth_getLogs / waitForTransactionReceipt calls via rpc.drpc.testnet.arc.network. A few things that may be useful context:

On the -32004 trigger rate

-32004 Certificate not found is chronic on both the public RPC and the DRPC endpoint — we see it in our keeper logs roughly every 3–5 minutes during normal operation at ~10 req/s. The key observation is it's usually harmless because the retry succeeds on the next peer. But the timing window your trace shows — where a 2-height range is batched right at the moment the first height's cert becomes available but the second doesn't yet — narrows the bug to a short race window. Given how frequently -32004 fires, hitting that window within hours is plausible.

On the three proposed fixes

Of your three proposals, (a) re-request the entire failed range is the most precise fix — it eliminates the gap at the source without relying on downstream detection. (b) gap detection (tip_height + 1 < min(pending_heights)) is worth implementing as a safety net regardless, since it would catch any future source of a skipped height. (c) pending-request timeouts would also help free the 5-slot deadlock as a last resort but doesn't address the gap itself.

The combination of (a) + (b) seems like the right approach: (a) prevents the gap at the source, (b) heals it if any other code path produces one.

Mitigation until the fix ships

Restart fully recovers the node (confirmed). A simple watchdog that detects consensus height stagnation works in the meantime:

#!/usr/bin/env bash
# Restart CL if consensus height hasn't advanced in 5 minutes
PREV=$(curl -sf http://localhost:31000/consensus/height | jq '.height')
sleep 300
CURR=$(curl -sf http://localhost:31000/consensus/height | jq '.height')
if [ "$CURR" -le "$PREV" ]; then
  echo "Consensus wedged at height $CURR — restarting CL"
  systemctl restart arc-node-consensus
fi

Run as a cron job every 5 minutes and it catches the wedge before it becomes an 18-hour gap.

On the EL parallel trie panic (possibly related)

The node at path Nibbles(0x71) does not exist panic in parallel.rs:3049 is a separate Reth upstream issue in the sparse parallel trie. The fact that it logged WARN Failed to compute state root in parallel and then recovered via the serial fallback is the intended degradation path — Reth's parallel trie is a performance optimization, not the primary path. The EL continuing normally after that warn is expected behavior. Worth filing upstream with Reth separately, but it's not part of the CL sync deadlock described here.

Version question

Is this known to be addressed in the upcoming v0.8.0? The dev-v0.8.0 branch appears to be in progress from the open PRs but there's no public release yet — asking in case a workaround note in the docs or a patch on v0.7.3 is feasible in the meantime.

[romac]

Thanks for flagging this and for the detailed report!

This is a known bug in the version of Malachite that Arc v0.7 uses. It’s already fixed upstream and will be rolled out in Arc v0.8.

[romac]

Let's leave this issue open until Arc v0.8 is out so anyone else running into it can find it easily.

[osr21]

Good to have the fix confirmed and the upstream Malachite connection identified — that explains why the -32004 window alone was enough to trigger a permanent wedge rather than a transient stall.

Two quick follow-ups if useful:

1. Upstream Malachite PR/commit reference?
If there's a specific PR or commit in the Malachite repo that contains the range-re-request fix, linking it here would help anyone building Arc from source know exactly which point on main includes the fix — and would give the community a place to review the fix approach (a vs b vs c from the original report).

2. Watchdog remains relevant until v0.8 ships
The cron-based height-stagnation watchdog in my earlier comment is a reasonable interim mitigation for anyone running follower nodes on v0.7.3 who can't wait for v0.8. Worth keeping that visible in the issue for operators who find this page in the meantime.

Thanks for the quick turnaround on confirmation.

[arjun215-eng]

Recurrence data: two more wedges on the same follower node (v0.7.3) within one day — 2026-07-22

Same signature both times as the original report (tip frozen, sync_height a few heights past tip_height, all 5 parallel request slots pinned, Maximum number of parallel requests reached, skipping request for values max_parallel_requests=5 pending_requests=5 spam until restart):

went quiet (UTC)	tip_height	sync_height	recovered
~13:27	53,088,995	53,089,002	CL restart @ 13:32:42
~23:02	53,157,165	53,157,172	CL restart @ 23:09:36

So under current testnet conditions this is hitting roughly twice a day, not a rare race — worth factoring into how urgently v0.8 reaches followers.

For anybody reading this before v0.8: the height-stagnation watchdog approach discussed above works reliably — both wedges fully recovered without a resync or snapshot needed. We've since tightened our check interval from 5 minutes to 1 minute (with a short post-restart grace period before re-checking), which caps the stall outage at ~1–2 minutes.

Happy to attach full log excerpts from either occurrence if useful for validating the fix.

[osr21]

The recurrence data is useful for the Arc team — twice a day is well into "reliably reproducible on demand" territory, not a rare race, which should make fix validation straightforward once v0.8 ships.

One pattern worth flagging across all three wedge observations (including the original report):

Event	tip_height	sync_height	gap
2026-07-21 original	52,893,873	52,893,880	7
2026-07-22 ~13:27	53,088,995	53,089,002	7
2026-07-22 ~23:02	53,157,165	53,157,172	7

The gap is consistently exactly 7 across all three: max_parallel_requests (5) + failed batch width (2) = 7. This is the expected mechanical outcome when the failed range covers exactly 2 heights — sync advances sync_height past both, retries only the first, then fills all 5 parallel slots with the following heights.

A few things this is useful for:

Watchdog precision. Height stagnation alone (tip hasn't moved in N minutes) is the right primary trigger, but adding a secondary check — sync_height - tip_height >= 6 at stagnation time — immediately distinguishes this specific bug from a general slow-sync or peer-connectivity issue. The latter would show a much smaller or larger gap. The 1-minute check interval you've settled on is tight enough that this combined check could drop the false-positive restart rate if you're seeing any.

Log excerpt value. If the gap in the new log excerpts is also exactly 7 in both cases, that confirms the failing batch is always 2 heights wide under current testnet conditions — consistent with the original sequence. If either shows a gap of 8 or 9, the batch width is occasionally 3+, which would mean the fix also needs to handle wider ranges (and is worth the Arc team knowing before v0.8 cut). That's the specific thing worth checking in those excerpts beyond confirming the bug recurred.

EL cleanliness across all three. All three recoveries were pure CL restarts — no EL intervention, no resync, no snapshot. That consistency confirms the wedge state is entirely in the CL's in-memory sync machine. EL state is intact throughout, which is also a useful fix-validation criterion: post-v0.8, a node that previously would have wedged here should simply continue advancing without any intervention, and the EL should see no disruption.

[arjun215-eng]

Interim mitigation result: BATCH_SIZE=1 (single-height sync requests) — 0 wedges in 41h+ on a previously-2x/day follower

Following up on my earlier data point with a concrete mitigation and its results, in case it's useful to other operators running v0.7.3 followers.

What we changed. Since the range-split can only skip a height when a multi-height range fetch partially fails, we pinned the sync batch size to 1 so every request covers exactly one height — a fetch then either delivers its height or fails whole and is re-requested; there's no partial case to lose a height in. On v0.7.3 that's a one-line change to malachite-app's hardcoded sync config (BATCH_SIZE: 10 → 1).

Deployed build (v0.7.3 + this patch, CL only, EL stock), with the reasoning and the residual trigger classes: https://github.com/arjun215-eng/arc-node/tree/v0.7.3-single-height-sync

Results (~41.4h after deploy, follower on Arc Testnet):

	Stock v0.7.3 (baseline)	With BATCH_SIZE=1
Wedges	~2/day (2 in the 24h before the patch)	0 in 41h+
CL restarts needed	per wedge	0 — single continuous process
Sync lag vs public tip	—	0, held across ~137k blocks/day

Cost: single-height fetches mean ~10x more RPC round trips during catch-up (2 per block instead of ~2 per 10-block batch). On a follower this was invisible — lag stayed pinned at 0 and it never tripped provider rate limits in steady state. It would matter more for a node doing a long cold catch-up.

Caveats. This only removes the range-split trigger. As noted in the writeup, a dead pending-request entry can in principle still form via other paths (a fetched value dropped from a full buffer, a processing-error attribution miss, timeout/response races) — none observed here, but the terminal signature would be identical, so we kept a height-stagnation watchdog running alongside regardless. This is why the tip_height + 1 < min(pending_heights) gap-detection safety net discussed earlier would be valuable upstream even with the range re-request fix in place.

[osr21]

Useful data — 41h+ clean is a meaningful baseline given the previous 2x/day rate.

A few notes from looking at the fork diff:

Watchdog simplification on the patched build

The gap-fingerprint secondary check I mentioned in an earlier comment — sync_height - tip_height >= 6 at stagnation time — is now structurally irrelevant on BATCH_SIZE=1. That specific signature requires a multi-height range to partially fail, which can't happen when every request covers exactly one height. Operators running this patch can simplify the watchdog back to pure height-stagnation detection and drop the gap check:

if [ "$CURR" -le "$PREV" ]; then
  systemctl restart arc-node-consensus
fi

The gap check is still worth keeping on stock v0.7.3 if distinguishing "this specific bug" from "general slow-sync" matters for alerting. On the patched build it adds noise.

The watchdog itself is still worth running alongside the patch, for the residual paths you flagged. If one of those alternate sources fires, the gap signature would differ from 7 (no range-split means no fixed-width hole before the 5-slot fill), which would itself be a useful diagnostic signal for the Arc team — a wedge with sync_height - tip_height != 7 post-patch would point to one of the non-range-split paths rather than a regression.

The git_branch addition

The fork also exposes git_branch in the /version RPC response (alongside the existing git_commit, git_short_hash, cargo_version). Small thing, but it means a quick GET /version confirms which build is active — "git_branch": "v0.7.3-single-height-sync" vs "main" vs a tag checkout — without needing to inspect binary metadata or check the systemd unit. Useful when coordinating with a monitoring system or confirming a rollout across multiple hosts.

On the July 21 recovery

The SYNC_WEDGE_ANALYSIS.md notes the first incident used a snapshot restore rather than a plain restart. That makes sense for an 18h gap (~100k+ blocks at Arc's ~1.6 block/s rate) — restart is always sufficient to unblock consensus, but snapshot avoids the catch-up time when the lag is large. The distinction is worth noting for anyone hitting this for the first time: CL restart is the minimal fix; snapshot is optional if the downtime was long enough that re-syncing from scratch is preferable.

[danilaverbena]

This looks like it's already fixed upstream in the malachite fork, just not yet pulled into arc-node's pin.

Root cause lives in crates/sync/src/handle.rs. When a batched fetch fails, re_request_values_from_peer_except rewinds sync_height back toward the start of the failed range so the untried heights can be re-requested — but the unconditional forward set at the end of send_and_track_request_to_peer:

// Update sync_height to the next uncovered height after this range
set_sync_height(state, final_range.end().increment());

clobbers that rewind. The stranded tail of the range ends up below sync_height and outside pending_requests, so request_values never re-issues it, the parallel slots fill with later heights, and the follower wedges — exactly the 52893874-fell-through-the-gap sequence in the report.

The fix is already merged: circlefin/malachite #1567 — fix(sync): preserve rewound sync_height after concurrent re-request (merged 2026-06-22) makes that forward set_sync_height conditional so it no longer overwrites a rewind. Two related sync-robustness fixes also landed after the current pin and are worth getting in the same bump: #1543 (reject non-contiguous sync responses) and #1556 (ignore stale invalid-value responses).

Why v0.7.3 still wedges: arc-node currently pins malachite at rev = 8ee5d998… (2026-05-22), which predates all three of those PRs. So the fix exists on main but isn't in the built binary.

Suggested resolution: bump the circlefin/malachite git dependency to a commit that includes #1567 (and ideally #1543 / #1556), then re-pin. Happy to open that dependency-bump PR if it's useful, though I suspect the periodic fork-sync process may already cover it — flagging so the gap is visible. (Separately, the reth parallel-state-root panic mentioned at the end looks unrelated and would be worth its own issue.)