Skip to content

perf(rpc): cache block traces#7108

Merged
hanabi1224 merged 3 commits into
mainfrom
hm/cache-trace
May 26, 2026
Merged

perf(rpc): cache block traces#7108
hanabi1224 merged 3 commits into
mainfrom
hm/cache-trace

Conversation

@hanabi1224
Copy link
Copy Markdown
Contributor

@hanabi1224 hanabi1224 commented May 26, 2026

Summary of changes

Changes introduced in this pull request:

  • add cache for StateManager::execution_trace
  • prefill cache on updating chain head

Benchmark

# command
oha -n 1000 -c 1 -m POST http://localhost:2345/rpc/v1 -H 'Content-Type: application/json' -D trace_block_call.json

# payload
{
	"method": "trace_block",
	"params": [
		"pending"
	],
	"id": 328,
	"jsonrpc": "2.0"
}

# pr
  Success rate: 100.00%
  Total:        4852.2867 ms
  Slowest:      17.3139 ms
  Fastest:      1.6483 ms
  Average:      4.8450 ms
  Requests/sec: 206.0884

  Total data:   28.94 MiB
  Size/request: 29.63 KiB
  Size/sec:     5.96 MiB

Response time histogram:
   1.648 ms [1]   |
   3.215 ms [361] |■■■■■■■■■■■■■■■■■■■■■■■■■■■
   4.781 ms [77]  |■■■■■
   6.348 ms [414] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
   7.915 ms [42]  |■■■
   9.481 ms [21]  |■
  11.048 ms [37]  |■■
  12.614 ms [28]  |■■
  14.181 ms [13]  |■
  15.747 ms [4]   |
  17.314 ms [2]   |
  
# main  
  Success rate:m100.00%   4m    5m    6m    7m    8m       ││0.0333  0.0752  0.1171  0.1590  0.2010  0.2429  0.2848    │
  Total:────────68138.6167 ms──────────────────────────────┘└──────────────────────────────────────────────────────────┘
  Slowest:      284.7552 ms
  Fastest:      33.0087 ms
  Average:      68.1236 ms
  Requests/sec: 14.6760

  Total data:   34.62 MiB
  Size/request: 35.46 KiB
  Size/sec:     520.34 KiB

Response time histogram:
   33.009 ms [1]   |
   58.183 ms [623] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
   83.358 ms [143] |■■■■■■■
  108.533 ms [77]  |■■■
  133.707 ms [108] |■■■■■
  158.882 ms [0]   |
  184.057 ms [1]   |
  209.231 ms [32]  |■
  234.406 ms [14]  |
  259.581 ms [0]   |
  284.755 ms [1]   |

Response time distribution:
  10.00% in 41.2875 ms
  25.00% in 43.0193 ms
  50.00% in 51.2385 ms
  75.00% in 66.9070 ms
  90.00% in 114.8404 ms
  95.00% in 132.5903 ms
  99.00% in 210.3713 ms
  99.90% in 284.7552 ms
  99.99% in 284.7552 ms

Reference issue to close (if applicable)

Closes

Other information and links

Change checklist

  • I have performed a self-review of my own code,
  • I have made corresponding changes to the documentation. All new code adheres to the team's documentation standards,
  • I have added tests that prove my fix is effective or that my feature works (if possible),
  • I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

Outside contributions

  • I have read and agree to the CONTRIBUTING document.
  • I have read and agree to the AI Policy document. I understand that failure to comply with the guidelines will lead to rejection of the pull request.

Summary by CodeRabbit

  • Performance Improvements

    • Caches execution traces for validated tipsets with async warm-up to reduce recomputation and improve responsiveness.
    • Adds heap-size tracking for trace types and message raw bytes to improve memory estimation.
  • Bug Fixes

    • Trace processing updated to safely share trace data during debugging and tracing.
    • Warm-up trace failures are logged as non-fatal warnings so validation continues without blocking.

Review Change Stack

@hanabi1224 hanabi1224 added the RPC requires calibnet RPC checks to run on CI label May 26, 2026
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 26, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: a7a39da2-1309-4744-b296-551d1264716d

📥 Commits

Reviewing files that changed from the base of the PR and between 83e59a9 and 1f9acb8.

📒 Files selected for processing (1)
  • src/state_manager/mod.rs

Walkthrough

StateManager caches per-tipset execution traces as Vec<Arc>; callers and trace entries updated to use Arc-wrapped results; RPC trace types derive GetSize with a centralized raw_bytes_heap_size_helper; daemon warmup now invokes execution_trace for validated tipsets.

Changes

Execution trace caching and Arc-wrapped results

Layer / File(s) Summary
StateManager trace cache infrastructure
src/state_manager/mod.rs, src/state_manager/execution.rs
Adds trace_cache: ForestCache<TipsetKey, (CidWrapper, Vec<Arc<ApiInvocResult>>)>, initializes it with name "tipset_trace", shallow-clones it, and implements cached execution_trace backed by execution_trace_inner returning Vec<Arc<ApiInvocResult>>.
Arc-wrapped ApiInvocResult propagation
src/rpc/methods/eth/trace/parity.rs, src/rpc/methods/eth.rs
TipsetTraceEntry.invoc_result now holds Arc<ApiInvocResult>; non_system_traces_with_positions accepts/returns Arc-wrapped items; debug_trace_transaction clones the selected execution_trace; unit test updated to build Arc<ApiInvocResult>.
RPC cache warmup integration
src/daemon/mod.rs
prefill_rpc_caches_for_tipset invokes StateManager::execution_trace for each newly validated tipset and logs a warning on failure without aborting warmup.

Heap-size tracking for trace types

Layer / File(s) Summary
Raw bytes heap-size helper
src/utils/get_size/mod.rs, src/shim/message.rs, src/shim/state_tree.rs
Adds raw_bytes_heap_size_helper for fvm_ipld_encoding::RawBytes; replaces local helper usage in shim::message::Message and adds impl GetSize for ActorState using big_int_heap_size_helper.
GetSize trait integration for RPC trace types
src/rpc/methods/state/types.rs
Trace-related types (e.g., ApiInvocResult, MessageGasCost, TraceIpld, ExecutionTrace, MessageTrace, ReturnTrace, GasTrace) derive GetSize. Field-level attributes ignore some fields (e.g., msg_cid, execution_trace, message, exit_code) and use raw_bytes_heap_size_helper for raw byte fields (params, cid, r#return). Imports reorganized to include GetSize and the helper.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

Suggested reviewers

  • LesnyRumcajs
  • sudo-shashank
  • akaladarshi
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 52.94% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'perf(rpc): cache block traces' directly and clearly describes the main change: adding caching for RPC block traces to improve performance.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch hm/cache-trace
✨ Simplify code
  • Create PR with simplified code
  • Commit simplified code in branch hm/cache-trace

Comment @coderabbitai help to get the list of available commands and usage tips.

@hanabi1224 hanabi1224 marked this pull request as ready for review May 26, 2026 14:34
@hanabi1224 hanabi1224 requested a review from a team as a code owner May 26, 2026 14:34
@hanabi1224 hanabi1224 requested review from LesnyRumcajs and sudo-shashank and removed request for a team May 26, 2026 14:34
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/daemon/mod.rs`:
- Around line 403-407: The synchronous call to
StateManager::execution_trace(&ts) inside async fn prefill_rpc_caches_for_tipset
can block Tokio worker threads; wrap that call in tokio::task::spawn_blocking
and await its JoinHandle result, propagating any error into the existing warn
log (i.e., replace the direct call in the block with a spawn_blocking closure
that calls state_manager.execution_trace(ts.clone() or move ts) and then handle
Err(e) from the join or the inner call the same way you currently log it);
ensure you move or clone ts and state_manager into the spawn_blocking closure as
needed and preserve the existing warn message when logging failures.

In `@src/rpc/methods/state/types.rs`:
- Around line 69-70: The field execution_trace on ApiInvocResult is currently
annotated with #[get_size(ignore)], causing GetSize to under-report memory;
remove that attribute so execution_trace is included in size accounting and
ensure ExecutionTrace implements GetSize (or provide a custom GetSize impl for
ApiInvocResult that accounts for execution_trace) so cached trace memory is
measured correctly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: a0575a7b-5bdc-4f09-ab90-e45b680aee39

📥 Commits

Reviewing files that changed from the base of the PR and between 32a6885 and f89858f.

📒 Files selected for processing (9)
  • src/daemon/mod.rs
  • src/rpc/methods/eth.rs
  • src/rpc/methods/eth/trace/parity.rs
  • src/rpc/methods/state/types.rs
  • src/shim/message.rs
  • src/shim/state_tree.rs
  • src/state_manager/execution.rs
  • src/state_manager/mod.rs
  • src/utils/get_size/mod.rs

Comment thread src/daemon/mod.rs
Comment thread src/rpc/methods/state/types.rs Outdated
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/utils/get_size/mod.rs (1)

62-67: ⚡ Quick win

Add rustdoc for the public heap-size helper.

Please add a doc comment on raw_bytes_heap_size_helper to document its approximation semantics and intended usage.

Suggested patch
+/// Estimates heap usage for [`fvm_ipld_encoding::RawBytes`].
+///
+/// This is a cheap approximation based on the underlying byte slice length,
+/// and may undercount compared with true allocation capacity.
 pub fn raw_bytes_heap_size_helper(b: &fvm_ipld_encoding::RawBytes) -> usize {
     // Note: this is a cheap but inaccurate estimation,
     // the correct implementation should be `Vec<u8>.from(b.clone()).get_heap_size()`,
     // or `b.bytes.get_heap_size()` if `bytes` is made public.
     b.bytes().get_heap_size()
 }

As per coding guidelines, "Document public functions and structs with doc comments".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/utils/get_size/mod.rs` around lines 62 - 67, Add a Rust doc comment to
the public function raw_bytes_heap_size_helper explaining that it returns a
cheap but approximate heap-size estimate by calling b.bytes().get_heap_size(),
describing the approximation semantics and limitations (i.e., it may be
inaccurate compared to converting to Vec<u8> or using bytes.get_heap_size() if
the inner bytes field were public), and stating the intended usage (quick
estimate only, not for exact accounting) along with any alternative recommended
approaches for precise measurement.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/utils/get_size/mod.rs`:
- Around line 62-67: Add a Rust doc comment to the public function
raw_bytes_heap_size_helper explaining that it returns a cheap but approximate
heap-size estimate by calling b.bytes().get_heap_size(), describing the
approximation semantics and limitations (i.e., it may be inaccurate compared to
converting to Vec<u8> or using bytes.get_heap_size() if the inner bytes field
were public), and stating the intended usage (quick estimate only, not for exact
accounting) along with any alternative recommended approaches for precise
measurement.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 0a1fa2be-fd42-45af-842c-94923d87f2bf

📥 Commits

Reviewing files that changed from the base of the PR and between f89858f and 4973f64.

📒 Files selected for processing (9)
  • src/daemon/mod.rs
  • src/rpc/methods/eth.rs
  • src/rpc/methods/eth/trace/parity.rs
  • src/rpc/methods/state/types.rs
  • src/shim/message.rs
  • src/shim/state_tree.rs
  • src/state_manager/execution.rs
  • src/state_manager/mod.rs
  • src/utils/get_size/mod.rs

@codecov
Copy link
Copy Markdown

codecov Bot commented May 26, 2026

Codecov Report

❌ Patch coverage is 71.42857% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.38%. Comparing base (32a6885) to head (1f9acb8).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/daemon/mod.rs 0.00% 3 Missing ⚠️
src/shim/state_tree.rs 0.00% 3 Missing ⚠️
src/utils/get_size/mod.rs 0.00% 3 Missing ⚠️
src/rpc/methods/eth.rs 66.66% 1 Missing and 1 partial ⚠️
src/state_manager/execution.rs 96.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
Files with missing lines Coverage Δ
src/rpc/methods/eth/trace/parity.rs 50.00% <ø> (-0.15%) ⬇️
src/rpc/methods/state/types.rs 100.00% <ø> (ø)
src/shim/message.rs 77.05% <ø> (+1.33%) ⬆️
src/state_manager/mod.rs 65.75% <100.00%> (+0.31%) ⬆️
src/state_manager/execution.rs 36.97% <96.00%> (+7.56%) ⬆️
src/rpc/methods/eth.rs 65.28% <66.66%> (-0.10%) ⬇️
src/daemon/mod.rs 25.68% <0.00%> (-0.12%) ⬇️
src/shim/state_tree.rs 58.50% <0.00%> (-0.61%) ⬇️
src/utils/get_size/mod.rs 60.00% <0.00%> (-6.67%) ⬇️

... and 8 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 32a6885...1f9acb8. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@hanabi1224 hanabi1224 added this pull request to the merge queue May 26, 2026
Merged via the queue into main with commit a3caf3e May 26, 2026
33 checks passed
@hanabi1224 hanabi1224 deleted the hm/cache-trace branch May 26, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

RPC requires calibnet RPC checks to run on CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants