-
Notifications
You must be signed in to change notification settings - Fork 406
feat(test-benchmark): updates and fixes for fixed opcode count #1985
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: forks/amsterdam
Are you sure you want to change the base?
feat(test-benchmark): updates and fixes for fixed opcode count #1985
Conversation
28a0c0c to
8305eab
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## forks/amsterdam #1985 +/- ##
================================================
Coverage 86.33% 86.33%
================================================
Files 538 538
Lines 34557 34557
Branches 3222 3222
================================================
Hits 29835 29835
Misses 4148 4148
Partials 574 574
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
8305eab to
08a6443
Compare
packages/testing/src/execution_testing/cli/pytest_commands/plugins/shared/benchmarking.py
Outdated
Show resolved
Hide resolved
LouisTsai-Csie
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some initial comments, will continue on more feedback:
- The required precompile benchmarks are already labeled with repricing marker, we only label these with necessary configurations. So we could remove the repricing marker in precompiles benchmark you add.
We need to update the benchmark_parser also, or it would override some of the setting. For example, this is valid in current logic: test_push* but it would be removed in the parser logic, as parser always find all the entry for the fixed-opcode-count compatible test.
Co-authored-by: 蔡佳誠 Louis Tsai <72684086+LouisTsai-Csie@users.noreply.github.com>
2e1a840 to
e75322c
Compare
f11e602 to
1104a8b
Compare
|
@LouisTsai-Csie I updated the parser and removed some precompiles: Modexp, BLS381, BN128 - as followed on discord. |
72063ae to
873f6d1
Compare
873f6d1 to
78d9b4a
Compare
🗒️ Description
This PR fixes several bugs and adds features for
--fixed-opcode-count.Features:
-m repricingmarker to all precompile tests, apart from select few not required or broken.test_modfrom repricing as not compatible with foc (fixed-opcode-count).OpcodeCountsConfigpattern matching and validation.0.25= 250 opcodes,0.5= 500 opcodes).Bug Fixes:
UsageErrorwhen the fixed opcode count config file (.fixed_opcode_counts.json) is missing.test_bitwise.*AND.*now correctly match against full test names instead oftest_bitwise.*only.More Details
Pattern matching now works with full test names, given both
"test_bitwise.*": [1]&"test_bitwise.*AND.*": [100]:test_bitwisevariants got 1K (pattern matched against bare function name).test_bitwise[fork_Prague-opcode_AND]gets 100K, others get 1K.For foc values < 1.0 (e.g., 0.25 = 250 opcodes), the inner iterations are set to the exact count with outer = 1, enabling precise low-count benchmarks.
🔗 Related Issues or PRs
Follow ups and fixes from #1790.
✅ Checklist
toxchecks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:uvx tox -e statictype(scope):.