Skip to content

fix(test-benchmark): correct compute benchmarks that ran below worst case#3136

Open
spencer-tb wants to merge 3 commits into
ethereum:devnets/glamsterdam/6-benchmarksfrom
spencer-tb:glam6-bench-degraded-signal
Open

fix(test-benchmark): correct compute benchmarks that ran below worst case#3136
spencer-tb wants to merge 3 commits into
ethereum:devnets/glamsterdam/6-benchmarksfrom
spencer-tb:glam6-bench-degraded-signal

Conversation

@spencer-tb

@spencer-tb spencer-tb commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Description

Follow-up to #3135. Where #3135 fixes benchmarks that measured nothing, this fixes three that measured the "wrong thing" where the gas is exact, but the work is below worst case.

  • test_arithmetic, drop the MOD/SMOD/ADDMOD/MULMOD cases. Their operand chain collapses to x mod 0 after ~150 iterations, so >99.99% of the loop is trivial (the params are even commented "not suitable ... values quickly become zero"). The dedicated test_mod / test_mod_arithmetic already benchmark these at worst case and carry the repricing marker, so these cases are redundant duplicates.
  • test_msize, drop mem_size=1_000_000. MSIZE is O(1), so its cost never varies with memory size. The large value only makes the one time quadratic memory expansion (~94% of the measured gas) dwarf the MSIZE loop, benchmarking expansion instead of MSIZE. The remaining sizes keep expansion ≤18%.
  • test_bls12_g1_msm / test_bls12_g2_msm, use distinct points and scalars. The input was a single point P1 repeated with scalar = the subgroup order r, so every term was r·P1 = O and an MSM can bucket the identical scalars / factor out the shared point. Distinct points paired with distinct full width scalars below r force the full k term computation. The exactly redundant k=128/k=64 cases in the test_bls12_381 sweep (which carry no repricing marker) are removed.

Related Issues or PRs

Follow-up to #3135, related to #3128.

Checklist

  • Ran fast static checks to avoid CI fails, see Code Standards & Verifying Changes: just static
  • PR title has the form <type>(<area>): <title>, where <type> and <area> come from an appropriate C-<type>, respectively A-<area>, label. The title should match the target squash commit message.

Cute Animal Picture

Put a link to a cute animal picture inside the parenthesis-->

@spencer-tb spencer-tb added C-bug Category: this is a bug, deviation, or other problem A-test-benchmark Area: execution_testing.benchmark and tests/benchmark labels Jul 9, 2026
@codecov

codecov Bot commented Jul 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (devnets/glamsterdam/6-benchmarks@eab26ab). Learn more about missing BASE report.

Additional details and impacted files
@@                         Coverage Diff                         @@
##             devnets/glamsterdam/6-benchmarks    #3136   +/-   ##
===================================================================
  Coverage                                    ?   93.30%           
===================================================================
  Files                                       ?      624           
  Lines                                       ?    36986           
  Branches                                    ?     3383           
===================================================================
  Hits                                        ?    34508           
  Misses                                      ?     1695           
  Partials                                    ?      783           
Flag Coverage Δ
unittests 93.30% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-test-benchmark Area: execution_testing.benchmark and tests/benchmark C-bug Category: this is a bug, deviation, or other problem

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant