feat(evm): EVM RPC .io/.iox integration tests (spec fixtures)#2985
feat(evm): EVM RPC .io/.iox integration tests (spec fixtures)#2985mojtaba-esk wants to merge 18 commits intomainfrom
Conversation
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2985 +/- ##
==========================================
+ Coverage 58.17% 58.19% +0.02%
==========================================
Files 2113 2112 -1
Lines 173688 173271 -417
==========================================
- Hits 101037 100842 -195
+ Misses 63622 63501 -121
+ Partials 9029 8928 -101
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
masih
left a comment
masih
left a comment
There was a problem hiding this comment.
Awesome! glad to see so many tests. This is going to up our game in terms of EVM API compatibility.
Question: of the 255 tests how many actually pass and what is the plan for those that do not?
Thanks. Here is a report on how many tests are used from ETH vanilla, how many modified slightly to match the data (blockhash, tx,...) and how many are actually generated copying the same style: https://github.com/sei-protocol/sei-chain/blob/8da453b5115ada53ffa0c9ece0fd86dc189e72de/integration_test/evm_module/rpc_io_test/RPC_IO_README.md 20 endpoints are not covered by Eth Exec API so we generated tests for them + 18 endpoints which we updated .io files to .iox (extended io) which has more stuff to handle some stateful test cases |
masih
left a comment
masih
left a comment
There was a problem hiding this comment.
The very beginning of hardened EVM compatible APIs, great work @mojtaba-esk 🚀
| continue | ||
| } | ||
|
|
||
| if inBinding && lastIdx >= 0 && strings.HasPrefix(trimmed, directivePrefix) { |
There was a problem hiding this comment.
Small point that parseIOFile doesn't seem to return error for invalid fixture directives. A malformed directive would then result in assertions getting dropped scilently.
There was a problem hiding this comment.
oops, correct. nice one. will add some validations to it.
|
|
||
| # EVM execution-apis .io is allowed to fail until endpoint coverage is fixed (non-blocking for PR merge) | ||
| - name: ${{ matrix.test.name }} | ||
| continue-on-error: ${{ matrix.test.name == 'EVM RPC .io/.iox (spec fixtures)' }} |
There was a problem hiding this comment.
Have you had a chance to dig through the ones that are failing to see why? it would be awesome to get them all to pass. Exiting to see so many passing already 🙌
There was a problem hiding this comment.
Not yet, but plan to do it.
| **Debug one or a few SEED tests:** Run only specific files with extra per-pair logging (request after substitution, bindings, whether `result.transactions` is present): | ||
|
|
||
| ```bash | ||
| SEI_EVM_IO_DEBUG_FILES="debug_getRawTransaction/get-tx.iox" go test ./integration_test/evm_module/rpc_io_test/ -v -run TestEVMRPCSpec |
There was a problem hiding this comment.
Also mention SEI_EVM_IO_RUN_INTEGRATION=1?
There was a problem hiding this comment.
sure, missed here.
3 participants
Describe your changes and provide context
Summary
This PR proposes to add a runner that executes execution-apis-style .io/.iox fixtures (255 tests) against Sei EVM RPC with spec-only checks. Script seeds block and deploy tx; fixtures use
__SEED__and bindings (${txHash},${deployTxHash}).Also a job ( "EVM RPC .io/.iox (spec fixtures)") is added to integration CI with
continue-on-errorso PRs can merge while endpoint coverage is incomplete; failures still show in the run.Why
Testing performed to validate your change
See the RPC_IO_README.md file for more details and the latest test run