Skip to content

Conversation

@spencer-tb
Copy link
Contributor

@spencer-tb spencer-tb commented Jan 7, 2026

🗒️ Description

This PR fixes several bugs and adds features for --fixed-opcode-count.

Features:

  • Use longest match for pattern specificity with regex, more detail below.
  • Add -m repricing marker to all precompile tests, apart from select few not required or broken.
  • Remove test_mod from repricing as not compatible with foc (fixed-opcode-count).
  • Adds unit tests for OpcodeCountsConfig pattern matching and validation.
  • Support float values for sub-1K opcode counts (e.g., 0.25 = 250 opcodes, 0.5 = 500 opcodes).

Bug Fixes:

  • Raise UsageError when the fixed opcode count config file (.fixed_opcode_counts.json) is missing.
  • Validate flag input to reject test paths accidentally consumed by argparse.
  • Major - Fix config file pattern matching for parametrized tests, patterns like test_bitwise.*AND.* now correctly match against full test names instead of test_bitwise.* only.

More Details

Pattern matching now works with full test names, given both "test_bitwise.*": [1] & "test_bitwise.*AND.*": [100]:

  • Before: All test_bitwise variants got 1K (pattern matched against bare function name).
  • After: test_bitwise[fork_Prague-opcode_AND] gets 100K, others get 1K.

For foc values < 1.0 (e.g., 0.25 = 250 opcodes), the inner iterations are set to the exact count with outer = 1, enabling precise low-count benchmarks.

🔗 Related Issues or PRs

Follow ups and fixes from #1790.

✅ Checklist

  • All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:
    uvx tox -e static
  • All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
  • All: Considered adding an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • All: Set appropriate labels for the changes (only maintainers can apply labels).

@spencer-tb spencer-tb added C-bug Category: this is a bug, deviation, or other problem C-feat Category: an improvement or new feature P-high A-test-benchmark Area: execution_testing.benchmark and tests/benchmark labels Jan 7, 2026
@spencer-tb spencer-tb force-pushed the feat/fixed-opcode-count-updates branch 2 times, most recently from 28a0c0c to 8305eab Compare January 7, 2026 15:19
@codecov
Copy link

codecov bot commented Jan 7, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.33%. Comparing base (b3543e9) to head (955ac67).

Additional details and impacted files
@@               Coverage Diff                @@
##           forks/amsterdam    #1985   +/-   ##
================================================
  Coverage            86.33%   86.33%           
================================================
  Files                  538      538           
  Lines                34557    34557           
  Branches              3222     3222           
================================================
  Hits                 29835    29835           
  Misses                4148     4148           
  Partials               574      574           
Flag Coverage Δ
unittests 86.33% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@spencer-tb spencer-tb force-pushed the feat/fixed-opcode-count-updates branch from 8305eab to 08a6443 Compare January 7, 2026 21:31
Copy link
Collaborator

@LouisTsai-Csie LouisTsai-Csie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some initial comments, will continue on more feedback:

  • The required precompile benchmarks are already labeled with repricing marker, we only label these with necessary configurations. So we could remove the repricing marker in precompiles benchmark you add.

We need to update the benchmark_parser also, or it would override some of the setting. For example, this is valid in current logic: test_push* but it would be removed in the parser logic, as parser always find all the entry for the fixed-opcode-count compatible test.

@spencer-tb spencer-tb force-pushed the feat/fixed-opcode-count-updates branch from 2e1a840 to e75322c Compare January 8, 2026 11:42
@spencer-tb spencer-tb force-pushed the feat/fixed-opcode-count-updates branch from f11e602 to 1104a8b Compare January 8, 2026 12:37
@spencer-tb
Copy link
Contributor Author

@LouisTsai-Csie I updated the parser and removed some precompiles: Modexp, BLS381, BN128 - as followed on discord.

@spencer-tb spencer-tb force-pushed the feat/fixed-opcode-count-updates branch from 72063ae to 873f6d1 Compare January 8, 2026 16:04
@spencer-tb spencer-tb force-pushed the feat/fixed-opcode-count-updates branch from 873f6d1 to 78d9b4a Compare January 8, 2026 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-test-benchmark Area: execution_testing.benchmark and tests/benchmark C-bug Category: this is a bug, deviation, or other problem C-feat Category: an improvement or new feature P-high

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants