core: agree on gloas self-built blocks without payload

[KaloyanTanev]

From the gloas fork, consensus agrees on self-built blocks without their execution payload envelope, blobs and KZG proofs, keeping megabytes out of every consensus message. The fetcher hands the contents to PayloadEx as this node's candidate and proposes the bare block with PayloadContentsRoot, a commitment to them. Once the block is agreed, the node whose candidate matches the agreed root stores the contents in the DutyDB and pushes them to its peers over /charon/payloadex/1.0.0; a peer that doesn't receive them within 1s asks its peers, which answer from their DutyDB as soon as they hold them. The DutyDB is the only store of the contents and accepts those hashing to the agreed root only, whoever delivers them. VCs are served the agreed block with Eth-Execution-Payload-Included: false and retrieve the envelope from the VAPI, which awaits the shared contents.

category: feature
ticket: #4324

[codecov]

Codecov Report

❌ Patch coverage is 70.06803% with 220 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (gloas@21040aa). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
core/corepb/v1/payloadex.pb.go	0.00%	64 Missing ⚠️
core/interfaces.go	0.00%	55 Missing ⚠️
core/payloadex/payloadex.go	86.74%	33 Missing ⚠️
core/validatorapi/validatorapi.go	82.27%	14 Missing ⚠️
core/dutydb/memory.go	86.07%	11 Missing ⚠️
core/tracing.go	0.00%	11 Missing ⚠️
app/app.go	0.00%	9 Missing ⚠️
core/ssz.go	82.35%	9 Missing ⚠️
core/deadline.go	91.22%	5 Missing ⚠️
core/fetcher/fetcher.go	66.66%	5 Missing ⚠️
... and 1 more

Additional details and impacted files

@@           Coverage Diff            @@
##             gloas    #4771   +/-   ##
========================================
  Coverage         ?   67.14%           
========================================
  Files            ?      252           
  Lines            ?    32954           
  Branches         ?        0           
========================================
  Hits             ?    22128           
  Misses           ?    10825           
  Partials         ?        1           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

🟡 Changes recommended

Unbounded incoming payload retention, premature exchange expiry, and incompatible payload-inclusion responses remain unresolved.

3 open findings

• Unbounded detached push handlers enable per-peer memory exhaustion · New
• Proposer deadline and retention prevent envelope recovery · New
• Self-built blocks omit contents when payload inclusion is requested · New

What changed in this PR

Moves Gloas self-built payload contents out of Charon’s consensus messages and distributes them separately after agreement.

Changes:

• Adds payload commitments and peer-to-peer push/pull exchange.
• Connects payload storage and retrieval to DutyDB and the validator API.
• Updates tests, tracing, metrics, and architecture documentation.

File	Description
testutil/​validatormock/​propose.go	Retrieves and signs separate envelopes.
testutil/​random.go	Generates self-built block fixtures.
docs/​metrics.md	Documents exchange metrics.
docs/​architecture.md	Describes the separate payload workflow.
core/​validatorapi/​validatorapi.go	Serves bare proposals and shared envelopes.
core/​validatorapi/​validatorapi_test.go	Tests separate envelope handling.
core/​unsigneddata.go	Adds payload commitments and stripping.
core/​unsigneddata_test.go	Tests commitments and serialization.
core/​tracing.go	Traces exchange and payload retrieval.
core/​ssz.go	Encodes proposal commitments.
core/​signeddata.go	Adds the JSON commitment field.
core/​payloadex/​payloadex.go	Implements payload push/pull exchange.
core/​payloadex/​payloadex_test.go	Tests exchange between peers.
core/​payloadex/​payloadex_internal_test.go	Tests candidates and message handling.
core/​payloadex/​metrics.go	Defines exchange counters.
core/​interfaces.go	Wires exchange and storage callbacks.
core/​fetcher/​fetcher.go	Separates payloads before consensus.
core/​fetcher/​fetcher_test.go	Tests payload separation.
core/​dutydb/​memory.go	Stores and retrieves committed contents.
core/​dutydb/​memory_test.go	Tests payload storage and expiry.
core/​dutydb/​memory_internal_test.go	Updates proposal query calls.
core/​corepb/​v1/​payloadex.proto	Defines exchange messages.
core/​corepb/​v1/​payloadex.pb.go	Adds generated message bindings.
app/​app.go	Initializes exchange and registers protocols.

Files not reviewed (1)

• core/corepb/v1/payloadex.pb.go: Generated file

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Payload availability, response validation, request-size limits, and stateless API compatibility need correction.

4 open findings

• Unbounded request reads allow excessive memory consumption · New
• Losing the winning fetcher prevents payload revelation · New
• Accepting responses for mismatched duties cancels valid requests · New
• Missing self-built payloads incorrectly return successful responses · New

3 resolved since last review

• Unbounded detached push handlers enable per-peer memory exhaustion
• Self-built blocks omit contents when payload inclusion is requested
• Proposer deadline and retention prevent envelope recovery

Files not reviewed (1)

• core/corepb/v1/payloadex.pb.go: Generated file

🧠 Review effort: Balanced

[OisinKyne]

4 sounds worth considering and what i was trying to get my head around in slack, will come back there. g2g for now

1. Stateless VC fallback is off-spec. Answer this before the fork. block.v4.yaml says a self-built block requested with include_payload=true carries the full contents. After 500ms the PR serves the bare block instead. A stateless VC may read "payload excluded" as "a builder won" and never fetch or sign the envelope, which loses the payload. Two things to do:

   • Test the VC matrix: Lighthouse, Teku, Lodestar, Prysm, Nimbus.
   • Until that's done, prefer waiting longer and staying on-spec.

   The reversed guidance also needs rolling out: until now we told operators to enable stateless, and the spec names DVs as the stateless use case. The DVN repos have no stateless flags set, so no compose changes are needed.

2. Envelope partial signatures still carry blobs. SubmitExecutionPayloadEnvelope still builds the partial signature with Blobs/KZGProofs. That makes it n�(n�1)�MB, now the largest message in the flow. Every node now holds verified contents in the DutyDB, so the partial can be signature plus root, rebuilt from the DutyDB at aggregation. This is state-of-play.md #3 and should be the next PR.

3. New crash window. If the building node dies between the decision and its push, no node holds the contents and the payload is never revealed. Before, every node had them. The receive side already holds pushes until the local decision, so letting the leader push its candidate alongside its pre-prepare would:

   • close that window;
   • take the transfer off the envelope's critical path.

   The cost is bandwidth spent on the rare non-agreed candidate. That's acceptable now that prepare and commit messages are small.

4. Split the commitment into an envelope root and a blobs root, now. The framing is new and unreleased, so this is the cheapest time to change it. Separate roots allow sending the envelope first: VCs could start signing from about 300 KB instead of up to 4 MB, with blobs following, or going only to the nodes that publish.

5. Candidates are keyed by slot, and a second store errors. A fetch retry, and later Phase 1 leader-timed fetching (several fetches per slot), would fail with "already stored". Keying by (slot, contents root) fixes it.

6. Minor: the stateless-VC warning fires before the builder-bid check, so it logs on every bid proposal. There's also no latency metric for decision � contents held; I'd add one, since it's what decides items 3 and 4.

[KaloyanTanev]

1. I will definitely test that, but some VCs even now don't accept stateless, we return stateless to them and they are treating them fine, so my assumption is they can adjust. And I'd rather experiment and test with that as of now, comparing to the alternative of not serving a block at all.

2. I wouldn't so quickly jump to sharing hashes, because:
   a) parsigex is configured currently to send over full data + sig, changing that would require new version of parsigex, if we want do to it properly
   b) it would mean on the receiver's end we have to always fetch from our db -> hash our data -> compare the hashes
   c) if a node is restarted or looses its internal data for some reason, we cannot send the data to the BN even if we receive enough partial sigs
   Probably we can think of something if we see this becoming an issue, but I don't want to overwhelm the whole parsigex only for some potential performance gain.

3. I know but that's the overall plan... you give some in order to get some 😄

4. Probably worth to do it now, yeah. However, we will always be "stealing" bandwidth from some other validator-related duty with this builder-related duty, if we decide to send blobs later on.

5. Reasonable as well. IDK if we will ever hit this edge case in production environment, but the price of doing that isn't big.

6. For the warning, this is by design as we want to troubleshoot misconfigured VCs. Reasonable on the metrics. I'll add some.

[sonarqubecloud]

Quality Gate passed

Issues
14 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud