Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
24124a6
feat(tests): port the repeated ECRECOVER call fillers (#3590)
chfast Sep 22, 2026
f9bbb76
feat(tests): close out eip-7954 checklist (#3546)
LouisTsai-Csie Sep 22, 2026
37615c9
feat(tests): more amsterdam coverage gaps (#3627)
spencer-tb Sep 22, 2026
80354f6
feat(tests): EIP-8037 - no refill for failed creates onto alive accou…
spencer-tb Sep 22, 2026
0a53ad8
feat(tests): cover block state budget after authorization preparation…
spencer-tb Sep 22, 2026
4b43084
fix(spec-tests): validate BPO fixtures with the framework's blob sche…
danceratopz Sep 22, 2026
756dc47
chore(ci,releases): fill `tests@` releases through Amsterdam (#3632)
danceratopz Sep 22, 2026
d90f397
feat(tests): improve EIP-8038 coverage, checklist, and ref-spec pin (…
gurukamath Sep 22, 2026
f085d2c
fix(tests): un-skip remaining Amsterdam ported static tests and drop …
spencer-tb Sep 23, 2026
faf6637
fix(spec-specs): wipe pre-existing storage on contract creation from …
spencer-tb Sep 23, 2026
4a05b82
fix(tests): isolate the BAL factory create-chain collision pre-alloc …
fselmo Sep 24, 2026
e64b257
refactor(test-fixtures): stream fixture writes and merges through par…
LouisTsai-Csie Sep 24, 2026
fb7fc09
fix(test-client-clis): accept every opcode name geth's t8n reports (#…
fselmo Sep 25, 2026
57d4144
refactor(test-vm, tests): add `create_op` helper and drop `CREATE`/`C…
LouisTsai-Csie Sep 25, 2026
f541fc1
feat(test-benchmark): pre-fund CREATE targets and add randomized jump…
chfast Sep 25, 2026
f1d9a04
feat(test-benchmark): add TSTORE key-distribution benchmark (#3637)
chfast Sep 25, 2026
84e7d2c
perf(test-fixtures): measure block RLP size without encoding it (#3642)
LouisTsai-Csie Sep 25, 2026
86f1982
feat(test-benchmark): add gas-mutated point-evaluation benchmark (#3660)
chfast Sep 28, 2026
2b19bf5
feat(tests): cover a null EIP-8070 custodyColumns and repin to the cu…
spencer-tb Sep 28, 2026
3e17067
feat(tests): cover BAL same-index nonce keep and code replace (#3654)
shubham5080 Sep 28, 2026
05e14bd
refactor(spec-specs): remove unreachable except in calculate_memory_g…
kenny-ish Sep 29, 2026
ecb68f2
fix(test-consume): run and check nethtest block tests in consume dire…
CPerezz Sep 29, 2026
05fec9a
fix(test-fill): fail collection of tests without a spec fixture (#3672)
CPerezz Sep 29, 2026
56dbe51
feat(tests): cover EIP-2780 preparation halt after repeated and self-…
chfast Sep 29, 2026
404c242
fix(test-clis): cast binary_path to string to prevent shutil.which cr…
jevinjojo Sep 30, 2026
949856f
feat(tests): cover EIP-8037 partial-reservoir new-account charge and …
chfast Sep 30, 2026
fa02bbf
Merge upstream/forks/amsterdam into from-upstream
pdobacz Sep 30, 2026
46f052c
resolve the upstream merge conflicts
pdobacz Sep 30, 2026
191e623
propagate the upstream fork changes into the Monad forks
pdobacz Sep 30, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .agents/skills/eip-checklist/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,15 @@ precompile/ = EIP-7702 does not introduce a precompile

(trailing `/` marks entire category as N/A)

## Code Coverage Items

`general/code_coverage/missed_lines` claims that every remaining miss is acceptable; it is not a place to park gaps.

- Measure branches: `uv run fill --cov=ethereum --cov-branch --cov-report=term-missing <eip test dir> --until <Fork>`. A line report counts an `if` whose false path never ran as covered, and without `--until` a fork under development fills nothing. Before calling a miss uncovered, rerun it against every suite that can reach that code.
- A missed branch is a test to write. Only two kinds of miss are acceptable: code that runs only outside `fill` (such as block validation in `fork.py`, run by `just json-loader`), and code the spec makes unreachable.
- An unreachable claim cites the guard that rules the branch out and a construction that was tried and failed, ideally by a second person or agent. "Nothing on mainnet does this" is not a reason while custom predeploy code (`pre_alloc_mutable`) or a crafted transaction reaches the branch at fill time.
- The entry lists only what is still missed, why, and a re-verify command that has been run as written; never covered branches or test names, which go stale. Re-derive the entry whenever the code it reasons about changes.

## Completed Examples

Reference these for patterns:
Expand Down
55 changes: 47 additions & 8 deletions .agents/skills/enhance-ported-test/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -331,6 +331,14 @@ verifies anything. Improve coupling and observability:
init code of step 0 is the same family, one level down: there the *bytecode*
stopped matching the scenario its own Yul comment describes.

**Engine X packing can introduce an intentional collision into another test.**
Two tests that deploy identical creator bytecode derive the same CREATE address.
If one predeploys a collider there and the other requires it absent, their
pre-allocations must not share a packed genesis. Give the collision test a named
`@pytest.mark.pre_alloc_group(...)`, then fill both with
`--generate-all-formats` and verify Engine X matches its ordinary engine sibling.
Validated on the `stRevertTest` CREATE collision and CREATE-OOG pair.

### 9. Introduce variables that encode relationships
Whenever a literal carries intent or two literals are logically linked, lift them
into named variables that express the *relationship*, not just the value. E.g.
Expand Down Expand Up @@ -539,10 +547,17 @@ calculator whenever the tx has calldata, or the derived `executed` (and
the cap) overstate. Validated on `test_refund_suicide50procent_cap`.

**A CREATE address collision burns the child's gas allowance** (the
EIP-684 path): the withheld child grant is consumed, nothing is created,
and under EIP-8037 the new-account state charge is refunded. Useful to
build always-failing creator frames with predictable consumption.
Validated on `test_revert_depth_create_address_collision`.
EIP-684 path): the withheld child grant is consumed and nothing is
created. Under EIP-8037 a colliding target is alive (it has code or a
nonce), so no new-account state gas is charged there in the first place.
**The burn is only observable if some arm lets the creator survive it**:
size the creator's post-collision slack so that its 1/64 retention cannot
pay the next store but the whole slack could (starved arm), or so the
retention clears the EIP-2200 gate and pays it (covered arm). A creator
that dies in every arm rolls the burn back with itself and the test
passes even when the grant is returned. Validated on both
`test_revert_depth_create_address_collision` files (mutation: returning
the grant on collision flips the starved arms).

**Loop-to-depth-1024 cannot replace loop-to-OOG.** With 63/64
attenuation, reaching depth 1024 needs ~e^16 × the terminal gas — no
Expand All @@ -551,12 +566,36 @@ fixed named budget with per-gas-schedule-era pinned depth counts, each
shift explained (±1 frame ≈ 64·ln(cost ratio)). Validated on
`test_loop_calls_depth_then_revert`.

**Framework wart: the SSTORE dirty-rewrite composite prices 100 on every
fork**, but Constantinople/Petersburg charge 5,000 for a dirty re-store —
a derived budget that must survive pre-Istanbul forks needs an explicit
headroom constant for it (named, commented). Observed on
**The SSTORE dirty-rewrite composite tracks the pre-Berlin schedules.**
`Op.SSTORE(key_warm=True, original_value=0, current_value=0xFF,
new_value=1).gas_cost(fork)` prices 5,006 on ConstantinopleFix, 806 on
Istanbul and 106 from Berlin, so a derived budget needs no extra
headroom constant for those forks (an earlier note here claimed
otherwise; the padding it prescribed also masked a wrongful new-account
charge on Amsterdam). Checked on
`test_revert_depth_create_address_collision`'s ConstantinopleFix sweep.

**A starved arm must still reach the opcode under test.** When a
"RevertDepth" style filler proves that a completed nested CREATE is
rolled back by a later OOG, the starved grant has to cover the CREATE
(and whatever stores precede it) and fall short only on the frame's
last store. A grant of "half the creator's needs" dies at the CREATE's
own charge, so the arm's `NONEXISTENT`/`nonce=1` expectations hold
without anything having been created. Prove the arm reaches the opcode
with a throwaway `raise` at the top of the fork's `generic_create` (or
`generic_call`): every arm must hit it. Validated on
`test_revert_depth_create2_oog` and `test_revert_depth_create_oog`.

**A "must OOG" budget has to fund the counterfactual.** A test whose
observable is "the frame ran out of gas because of X" (a collision burn,
a failed sub-call) only pins X if the same budget would have *completed*
without X. Size the slack as `victims + what X would have cost had it
gone through + margin`, then guard that X's leavings (the 1/64
retention) cannot pay the victims. Check it by disabling X in a spec
copy (e.g. `account_deployable` returning True): the test must fail.
Validated on `test_create2collision_selfdestructed_oog`, whose 30k
slack had been below its victims on every fork.

**EIP-8037 repriced the code deposit's regular part — boundaries beware.**
On 8037 forks the deposit charges only the keccak word cost
(`OPCODE_KECCAK256_PER_WORD * ceil32(len)/32`, ~6 gas) as regular gas plus
Expand Down
6 changes: 3 additions & 3 deletions .github/scripts/resolve_cached_release.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@

Dispatching `release_fixtures.yaml` with the `cached` flag drafts a
`tests@` release from the newest nightly artifact instead of
refilling: the scheduled nightly runs already build the mainnet
`tests` feature into a release-shaped `fixtures_<commit>` artifact.
refilling: the scheduled nightly runs already build the `tests`
feature into a release-shaped `fixtures_<commit>` artifact.
The `commit` input picks the nightly built at that commit instead of
the newest one. This script validates the request, picks the nightly
run whose artifact the release job downloads, and pins the exact
Expand Down Expand Up @@ -118,7 +118,7 @@ def newest_tests_tag(repository: str) -> str:

The `tests@` ref prefix cannot match any other feature's tags
(those are namespaced `tests-<feature>@`), so every match is a
mainnet tests release.
`tests` release.

The listing is paginated in ref-name order, not version order
(`tests@v9...` sorts after `tests@v20...`), so every page must be
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/release_fixtures.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -113,8 +113,8 @@ jobs:
id: matrix
shell: bash
env:
# Scheduled runs have no inputs: fill the mainnet `tests`
# feature; the placeholder version passes validation and is
# Scheduled runs have no inputs: fill the `tests` feature;
# the placeholder version passes validation and is
# otherwise unused because the `release` job is skipped for
# scheduled runs.
INPUT_FEATURE: ${{ inputs.feature || 'tests' }}
Expand Down
2 changes: 1 addition & 1 deletion docs/dev/releasing_tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ The release is created as a draft; review and publish it from the GitHub release

## Nightly fill

The same workflow also runs on a nightly schedule (02:00 UTC) as a release rehearsal: it fills the mainnet `tests` feature (all tests, slow included, all fixture formats, up to the latest mainnet fork — dev forks are not included) through the exact release pipeline, but stops after `combine`, so no tag or release is created. Each run uploads a `fixtures_<commit>` workflow artifact (short hash of the built commit, containing `fixtures.tar.gz`) with a 5-day retention: a rotating, always-available build of the mainnet fixtures, effectively a `tests@` release candidate on demand. A scheduled run skips itself when there are no new commits since the last nightly that actually filled — a skipped or failed nightly never advances that baseline, so no commit slips through unfilled. A quiet stretch without commits still re-fills once the last fill is four days old, or its artifact is gone, so a live artifact always exists within the five-day retention.
The same workflow also runs on a nightly schedule (02:00 UTC) as a release rehearsal: it fills the `tests` feature (all tests, slow included, all fixture formats, per its `fill-params` in `feature.yaml`) through the exact release pipeline, but stops after `combine`, so no tag or release is created. Each run uploads a `fixtures_<commit>` workflow artifact (short hash of the built commit, containing `fixtures.tar.gz`) with a 5-day retention: a rotating, always-available build of the `tests` fixtures, effectively a `tests@` release candidate on demand. A scheduled run skips itself when there are no new commits since the last nightly that actually filled — a skipped or failed nightly never advances that baseline, so no commit slips through unfilled. A quiet stretch without commits still re-fills once the last fill is four days old, or its artifact is gone, so a live artifact always exists within the five-day retention.

## Cached releases

Expand Down
2 changes: 1 addition & 1 deletion docs/running_tests/consume/cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ A release specification has the format `<release_name>@<version>`.

**Supported release names:**

- `tests`: The mainnet release, all tests for all forks up to and including the latest mainnet fork. A bare `latest` or `vX.Y.Z` input is shorthand for `tests@latest`, respectively `tests@vX.Y.Z`.
- `tests`: The stable release for client CI, all tests for all forks up to and including the fork clients' master branches implement (see [Test Release Types](../releases.md#test-release-types)). A bare `latest` or `vX.Y.Z` input is shorthand for `tests@latest`, respectively `tests@vX.Y.Z`.
- `<feat>-devnet`: Devnet releases, e.g. `bal-devnet`, `glamsterdam-devnet`.
- Other features: e.g. `benchmark`, `zkevm`.

Expand Down
10 changes: 6 additions & 4 deletions docs/running_tests/releases.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,11 @@ and cadence.
| Devnet | `<feat>-devnet@vX.Y.Z` | `fixtures_<feat>-devnet.tar.gz` | All forks, all tests, for an upcoming-fork feature under active devnet testing | the devnet or EIP branch |
| Benchmark | `benchmark@vX.Y.Z` | `fixtures_benchmark.tar.gz` | EVM benchmarking tests | latest `forks/*` branch |

- "Tests" releases track clients' production branches and are tagged frequently (roughly
once or twice a week). They are the "must pass" release for mainnet CI, and supersede the
old `fixtures_stable` / `fixtures_develop` artifacts.
- "Tests" releases aim to match clients' master branches. Once clients have merged the
upcoming fork into those branches ahead of a pending client release, the `tests` release
includes that fork too, and `X` bumps to its fork number. They are the "must pass" release
for client CI, tagged frequently (roughly once or twice a week), and supersede the old
`fixtures_stable` / `fixtures_develop` artifacts.
- "Devnet" releases target a specific feature under active development (e.g. `bal-devnet`).
They are advisory/non-blocking and may not yet cover every EIP; see the corresponding
release notes for the coverage provided.
Expand Down Expand Up @@ -148,7 +150,7 @@ fixtures/

Mapped to a typical client CI setup:

- **Blocking gate (current + past forks)**: Pin a specific `tests@vX.Y.Z` for reproducible,
- **Blocking gate (the forks on your `master` branch)**: Pin a specific `tests@vX.Y.Z` for reproducible,
no-rug-pull CI on your `master`/production branch, or follow the latest `tests` release if
a moving target is acceptable. This supersedes the old `fixtures_develop` / `fixtures_stable`
artifacts.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Depending on the changes introduced by an EIP, the following template is the min

| ID | Description | Status | Tests |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------ | ----- |
| `general/code_coverage/eels` | Run produced tests against [EELS](https://github.com/ethereum/execution-specs) and verify that line code coverage of new added lines for the EIP is 100%, with only exceptions being unreachable code lines. | | |
| `general/code_coverage/eels` | Run produced tests against [EELS](https://github.com/ethereum/execution-specs) and verify that branch coverage of new added lines for the EIP is 100%, with only exceptions being unreachable code lines. | | |
| `general/code_coverage/test_coverage` | Run coverage on the test code itself (as a basic logic sanity check), i.e., `uv run fill --cov tests`. | | |
| `general/code_coverage/missed_lines` | Document any lines missed in coverage reports and explain why they are acceptable (e.g., unreachable code, general infrastructure not related to the EIP). | | |
| `general/code_coverage/second_client` | Optional - Run against a second client and verify sufficient code coverage over new code added for the EIP. | | |
Expand Down
2 changes: 2 additions & 0 deletions packages/testing/src/execution_testing/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,7 @@
OpcodeCallArg,
Opcodes,
call_return_code,
create_op,
)

__all__ = (
Expand Down Expand Up @@ -257,6 +258,7 @@
"compute_create_address",
"compute_create2_address",
"compute_deterministic_create2_address",
"create_op",
"Create2PreimageLayout",
"CreatePreimageLayout",
"extend_with_defaults",
Expand Down
11 changes: 10 additions & 1 deletion packages/testing/src/execution_testing/base_types/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,13 @@
from .conversions import to_bytes, to_hex
from .pydantic import CamelModel, EthereumTestBaseModel, EthereumTestRootModel
from .reference_spec import ReferenceSpec
from .serialization import RLPSerializable, SignableRLPSerializable
from .serialization import (
KnownEncodedSize,
RLPSerializable,
SignableRLPSerializable,
encoded_prefixed_size,
encoded_size,
)
from .typing_utils import unwrap_annotation

__all__ = (
Expand Down Expand Up @@ -76,8 +82,11 @@
"Number",
"NumberBoundTypeVar",
"ReferenceSpec",
"KnownEncodedSize",
"RLPSerializable",
"SignableRLPSerializable",
"encoded_prefixed_size",
"encoded_size",
"StateCommitment",
"Storage",
"StorageKey",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
"""Ethereum test types for serialization and encoding."""

from dataclasses import astuple, is_dataclass
from typing import Any, ClassVar, List, Self, Sequence

import ethereum_rlp as eth_rlp
from ethereum_types.numeric import Uint
from ethereum_rlp.exceptions import EncodingError
from ethereum_types.numeric import FixedUnsigned, Uint
from trie import HexaryTrie

from execution_testing.base_types import Bytes
Expand All @@ -26,6 +28,81 @@ def to_serializable_element(v: Any) -> Any:
raise Exception(f"Unable to serialize element {v} of type {type(v)}.")


class KnownEncodedSize:
"""Stand in for an item whose encoded size is already known."""

__slots__ = ("size",)

def __init__(self, size: int) -> None:
"""Record the encoded size of the item this stands in for."""
self.size = size


def encoded_prefixed_size(payload_size: int) -> int:
"""
Return the encoded size of an item with a payload of `payload_size`
bytes, its length prefix included.
"""
if payload_size < 0x38:
return 1 + payload_size
return 1 + len(Uint(payload_size).to_be_bytes()) + payload_size


def _encoded_bytes_size(raw_bytes: bytes | bytearray) -> int:
"""Return the length of `eth_rlp.encode_bytes(raw_bytes)`."""
if len(raw_bytes) == 1 and raw_bytes[0] < 0x80:
return 1
return encoded_prefixed_size(len(raw_bytes))


def _encoded_sequence_size(raw_sequence: Sequence[Any]) -> int:
"""Return the length of `eth_rlp.encode_sequence(raw_sequence)`."""
return encoded_prefixed_size(
sum(encoded_size(item) for item in raw_sequence)
)


def _encoded_unsigned_size(value: int) -> int:
"""Return the length of `eth_rlp.encode(Uint(value))`."""
if value < 0x80:
return 1
return encoded_prefixed_size((value.bit_length() + 7) // 8)


def encoded_size(raw_data: Any) -> int:
"""
Return `len(eth_rlp.encode(raw_data))` without building the encoding.

This covers the same cases as `ethereum_rlp.encode`, plus
`KnownEncodedSize` placeholders, so measuring the size of a large
structure costs a walk over it instead of megabytes of intermediate byte
strings. The concrete types come first: the abstract
`Sequence` check is slow enough to dominate the walk.
"""
if isinstance(raw_data, (bytearray, bytes)):
return _encoded_bytes_size(raw_data)
elif isinstance(raw_data, (list, tuple)):
return _encoded_sequence_size(raw_data)
elif isinstance(raw_data, (Uint, FixedUnsigned)):
return _encoded_unsigned_size(int(raw_data))
elif isinstance(raw_data, KnownEncodedSize):
return raw_data.size
elif isinstance(raw_data, bool):
return 1
elif isinstance(raw_data, str):
return _encoded_bytes_size(raw_data.encode())
elif isinstance(raw_data, Sequence):
return _encoded_sequence_size(raw_data)
elif is_dataclass(raw_data) and not isinstance(raw_data, type):
return _encoded_sequence_size(astuple(raw_data))
else:
raise EncodingError(
"RLP encoded size of type {} is not supported".format(
type(raw_data)
)
)


class RLPSerializable:
"""Class that adds RLP serialization to another class."""

Expand Down Expand Up @@ -154,6 +231,14 @@ def rlp(self) -> Bytes:
self.get_rlp_prefix() + eth_rlp.encode(self.to_list(signing=False))
)

def rlp_size(self) -> int:
"""Return `len(self.rlp())` without building the encoding."""
if self.rlp_override is not None:
return len(self.rlp_override)
return len(self.get_rlp_prefix()) + encoded_size(
self.to_list(signing=False)
)

@classmethod
def list_root(cls, element_list: Sequence[Self]) -> bytes:
"""Return the root of a list of the given type."""
Expand Down
Loading
Loading