bug: eth_getLogs on rpc.testnet.arc.network times out / fails silently for wide block ranges, breaking indexers and dApp analytics #83

Activity

1. osr21 commented on May 25, 2026

   

   osr21

   on May 25, 2026

   Author

   More actions

   Safe batch size confirmed: 10,000 blocks per request

   After testing, batching at 10,000 blocks reliably returns without timeout.

   async function getAllLogs(provider: ethers.Provider, filter: ethers.Filter) {
     const latest = await provider.getBlockNumber();
     const BATCH  = 10_000;
     const logs: ethers.Log[] = [];
   
     for (let from = 0; from <= latest; from += BATCH) {
       const chunk = await provider.getLogs({
         ...filter,
         fromBlock: from,
         toBlock:   Math.min(from + BATCH - 1, latest),
       });
       logs.push(...chunk);
     }
     return logs;
   }

   Note: Arc Testnet is currently at ~43 M+ blocks, so iterating from block 0 takes ~4,300 batches. For most practical use cases, it's better to start from a known deployment block rather than from 0.
2. ZhiyuCircle

   added

   bugSomething isn't workingSomething isn't working

   ELComponent: ELComponent: EL

   on Jul 23, 2026
3. huklaa commented on Sep 6, 2026

   

   huklaa

   on Sep 6, 2026

   More actions

   I'm working on this
4. osr21 commented on Sep 6, 2026

   

   osr21

   on Sep 6, 2026

   Author

   More actions

   Disclosure: I'm not affiliated with Circle — I'm an external community contributor, not a maintainer, and I have no write access to this repository. This is one contributor's technical assessment, advisory only; please defer to Circle maintainers.

   Thanks for picking this up, @huklaa. I filed this in May, so before you sink time into it I re-measured the current behaviour against rpc.testnet.arc.network and rpc.testnet.arc.io today (head ≈ 60,789,000). A lot of it is already fixed, one of my own follow-up recommendations is now wrong, and most of what's left may not live in this repo. Details so you can scope it.

   What's fixed: the silent hang is gone

   The original repro no longer hangs past 60 s. It now fails fast:

   eth_getLogs {address, fromBlock: 0x0, toBlock: "latest"}
     -> 400-1000 ms, JSON-RPC error  (never > 1.1 s over 12 consecutive runs)
   

   Ask #2 in the issue ("return a clear error instead of timing out") is substantially satisfied. If your plan was to fix the timeout, that work looks done.

   What's still wrong: five error shapes, two of them misleading, and it's nondeterministic

   Measured today, address-filtered, 0x4dc7A9BbcB1139cDeDf5274272F541461ef4d20E:

   request	result
   span = 30,000 blocks	OK
   span = 30,001 blocks	-32012 "requested range too large"
   span = 100,000 blocks	-32012 "requested range too large"
   fromBlock: 0x0, toBlock: "latest"	-32614 "eth_getLogs is limited to a 10,000 range"
   fromBlock: 0x0, toBlock: <hex head>	4444 "pruned history unavailable" or -32614 — see below
   fromBlock: "earliest", toBlock: <hex>	-32602 "invalid block range params"
   unfiltered, span = 1,000 (log-dense)	-32602 "request exceeded max allowed range: query exceeds max results 20000, retry with the range 60787879-60788280"
   bursts	-32005 "rate limit exceeded"

   Three things worth flagging:

   1. The advertised limit is 3× off the enforced one. The -32614 text says "limited to a 10,000 range", but the actual cap is exactly 30,000 blocks — 30,000 succeeds and 30,001 returns -32012, reproducibly. That also makes my own follow-up comment above ("safe batch size confirmed: 10,000") needlessly conservative: 30,000-block chunks work, which is 2,026 requests to cover the chain instead of 6,079. I'd rather correct that than leave it as the accepted answer in this thread.

   2. The same request returns different errors on repeat. Ten identical {address, fromBlock: 0x0, toBlock: <hex head>} calls:

   endpoint	4444 pruned history unavailable	-32614 limited to a 10,000 range
   rpc.testnet.arc.network	8 / 10	2 / 10
   rpc.testnet.arc.io	5 / 10	5 / 10

   So the endpoint fans out across backends that implement the limits differently. A client cannot branch on the error code — which undercuts ask #2 even though the timeout itself is fixed. Both hostnames behave identically otherwise (same caps, both server: cloudflare), so the .network / .io distinction isn't a factor.

   3. 4444 pruned history unavailable is inaccurate — logs are not pruned. This one surprised me, and it invalidates the framing of my own issue. Single-block eth_getLogs succeeds at blocks 1, 1,000, 1,000,000, 10M, 30M, 50M; a binary search over the whole chain puts the oldest queryable block at 1. Only block 0 errors.

   fromBlock: 0x0, toBlock: 0x2710  ->  4444 pruned history unavailable
   fromBlock: 0x1, toBlock: 0x2711  ->  OK, 3 logs        # same span, +1 block offset
   

   So full log history is available (chain spans 2025-09-11 → now, ~361 days, ~0.51 s block time) and the error is reporting unavailability for data that is there. The practical effect is that fromBlock: 0 — the most natural way to write a cold-start scan, and exactly what my repro used — fails with an error that tells you to give up, when fromBlock: 1 works. State is genuinely pruned at genesis (eth_getBalance at block 0 → -32603 "state at block #1 is pruned"), so the two may share a code path; logs shouldn't inherit it.

   Also worth noting 4444 isn't a JSON-RPC or EIP-1474 code, and -32614 isn't either.

   Where the fix can and can't live — check before you patch

   I'd verify which layer you own first, because I think only one of these errors originates in this repo:

   • -32602 ... query exceeds max results 20000, retry with the range X-Y is reth's own error. DEFAULT_MAX_LOGS_PER_RESPONSE = 20_000 and DEFAULT_MAX_BLOCKS_PER_FILTER = 100_000 in reth-rpc-server-types at tag v2.2.0, which this repo pins. Grepping the tree, arc-node sets neither --rpc.max-blocks-per-filter nor --rpc.max-logs-per-response anywhere, so both run at reth defaults. Notably this is also the only error of the five that tells the client how to recover, and it's the one the node produces unaided.
   • -32614, -32012, 4444, -32005, and the 30,000-block cap are all stricter than the reth defaults and don't appear anywhere in this repository. They're imposed by the gateway in front of the nodes. A change to arc-node will not alter what a dApp sees at the public endpoints.

   If you own the gateway config too, the highest-value fixes are probably: make the enforced cap and the error text agree, stop returning 4444 for a fromBlock: 0 query on non-pruned data, and settle on one code per condition so clients can branch.

   Ask #1 (documentation) is still fully open

   docs.arc.io/arc/references/rpc-endpoints now lists endpoints, chain ID 5042002, node providers, supported methods and a troubleshooting table — but it documents no eth_getLogs range limit, no result-size limit, no rate limits, and says nothing about archive availability. eth_getLogs isn't even in the "Supported methods" table; there's no Logs/Filters row at all, and the troubleshooting table covers only connection refused / wrong chain ID / insufficient funds.

   That said — per the scope note in this repo's docs, arc-node documentation is node-operator facing and dApp-facing docs live on the Arc docs site. So the documentation half of this issue likely isn't actionable here either, and may need to be routed rather than fixed.

   Practical guidance for anyone hitting this today

   • Chunk at 30,000 blocks when address-filtered, not 10,000.
   • Start at block 1, not 0.
   • Handle -32602 ... retry with the range X-Y by honouring the suggested range — that's the log-density cap (20,000 results), independent of the block-range cap, and it's the only self-correcting one.
   • Treat 4444 / -32614 / -32012 as one "narrow the range" condition, since which one you get is not deterministic.
   • Back off on -32005.

   One adjacent thing if you're touching this method anyway: eth_getLogs also mishandles the pending block tag — it resolves to head + 1, so identical back-to-back calls alternate between the next block's logs and -32602 "block range extends beyond current head block". That one is node-side, and it's under discussion in #340.

   Happy to re-run any of these probes against a candidate fix.
5. huklaa commented on Sep 6, 2026

   

   huklaa

   on Sep 6, 2026

   More actions

   Thanks for re-testing this and for the detailed breakdown. Given the current behavior, I’m going to step back from implementing a repo-side timeout fix here: the original silent hang appears resolved, and most of the remaining inconsistencies look to be imposed by the public RPC gateway rather than arc-node itself. I’ll avoid opening a patch that wouldn’t affect what dApps see at the public endpoints. If maintainers confirm a repo-owned piece of this issue that still needs a code change, I’m happy to pick that up.
6. osr21 commented on Sep 11, 2026

   

   osr21

   on Sep 11, 2026

   Author

   More actions

   Independently checked the two claims this decision rests on, plus one thing worth flagging before this is closed as gateway-only.

   Confirmed: the reth-default claim is accurate

   Cloned the repo and grepped for the two limiting flags:

   $ grep -rn "max-blocks-per-filter\|max-logs-per-response" --include=*.rs --include=*.toml .
   (no matches)
   

   Neither --rpc.max-blocks-per-filter nor --rpc.max-logs-per-response is set anywhere in this tree, so both run at reth's built-in defaults. Checked those defaults directly against the pinned tag (v2.2.0):

   crates/rpc/rpc-server-types/src/constants.rs:
     DEFAULT_MAX_BLOCKS_PER_FILTER: u64 = 100_000
     DEFAULT_MAX_LOGS_PER_RESPONSE: usize = 20_000
   

   Both match the numbers already quoted in this thread. So the -32602 ... query exceeds max results 20000 error is reth's own default, unmodified by this repo, and the 30,000-block cap / -32614 / -32012 / 4444 / -32005 behavior is stricter than anything reth ships or this repo configures — consistent with those living in the gateway rather than arc-node. That supports stepping back from a repo-side patch for those five error shapes.

   One thing that is still repo-owned and low-risk: operator docs for the two flags above

   crates/node/README.md and docs/running-an-arc-node.md document --rpc.max-connections, --rpc.max-subscriptions-per-connection, and --arc.rpc.max-batch-entries, but neither mentions --rpc.max-blocks-per-filter or --rpc.max-logs-per-response at all, even though the binary accepts both (inherited from reth's CLI). Adding a line for each — what they do and their default values — would be a documentation-only change, independent of the gateway question, and it partially answers ask #1 from the original report within the scope this repo's docs actually own (operator-facing, per the existing scope note). It would not change what a dApp sees at the public endpoint, since the gateway enforces its own separate, tighter limits regardless of these flags — but it removes ambiguity about what this repo itself does and doesn't cap.

   The one node-side bug in this thread already has a fix in flight

   The pending block-tag issue flagged as "node-side" here is PR #340 (fix: normalize pending block tags in eth_getLogs filters to latest). It's open, has a pushed fix plus test coverage, and has an external review approval already on it — just waiting on a maintainer to run CI and merge. So no separate pickup is needed for that piece either.

   Net: the decision to step back from the five gateway-shaped errors holds up under an independent check. The only concrete repo-owned action left on this issue is the two-line docs addition above, and it doesn't require resolving who owns the gateway first.

   Disclosure: external community contributor, not affiliated with Circle, no write access to this repository. Advisory only.