Skip to content

feat: add mev cli tests#300

Open
PoulavBhowmick03 wants to merge 10 commits intomainfrom
poulav/mev_test
Open

feat: add mev cli tests#300
PoulavBhowmick03 wants to merge 10 commits intomainfrom
poulav/mev_test

Conversation

@PoulavBhowmick03
Copy link
Copy Markdown
Collaborator

Closes #235

@iamquang95
Copy link
Copy Markdown
Collaborator

@claude

@claude
Copy link
Copy Markdown

claude bot commented Apr 2, 2026

Claude finished @iamquang95's task in 2m 58s —— View job


Code Review

  • Gather context and read changed files
  • Review crates/cli/src/commands/test/mev.rs
  • Review supporting changes
  • Post findings

Summary

This PR ports Charon's MEV relay test suite into the Pluto CLI, adding Ping, PingMeasure, and CreateBlock test cases plus two unit tests. The implementation is largely correct but has one functional parity gap (credential stripping), a concurrency correctness issue in timeout handling, and a structural refactor concern.


Findings

[High] mev_ping_measure_test ignores embedded URL credentials

  • mev.rs:319request_rtt is called with the raw target URL. But request_rtt (mod.rs:699) uses the URL as-is via reqwest, which does not handle user:pass@host in URLs in the same way a manual basic_auth call does.
  • mev_ping_test (line 287–294) correctly calls parse_endpoint_credentials and apply_basic_auth. mev_ping_measure_test skips this step entirely.
  • Impact: If a MEV relay endpoint contains embedded credentials (e.g. https://0x<pubkey>@relay.example.com), the PingMeasure test will either fail with a 401 or include credentials in the URL as plain text.
  • Fix this →

[Medium] Timeout in test_single_mev only emits one failure entry, not N

  • mev.rs:250–261 — On cancellation, only one TestResult::Fail is pushed for queued_tests[test_counter]. If multiple tests were still queued, the remaining ones (indices test_counter+1..) are silently dropped from the results.
  • Impact: Score calculation (via calculate_score) and result output will silently omit cancelled tests beyond the first, making the reported score look better than it is and breaking parity with the Go implementation which emits a failure for every remaining test.
  • Fix this →

[Medium] reqwest::Client is created per request, not reused

  • mev.rs:285, mev.rs:547, mev.rs:574, mev.rs:616 — A new reqwest::Client is instantiated for every HTTP call. The reqwest docs recommend creating a client once and reusing it, as it manages a connection pool internally.
  • Impact: Under a load test with --number-of-payloads > 1, connection pooling is never utilized; each round-trip re-establishes TCP connections. RTT measurements may be inflated by connection setup overhead, making results unreliable for CreateBlock.
  • Recommendation: Thread a shared reqwest::Client through TestMevArgs or pass it as a parameter to the helper functions.

[Low] wiremock removed from dev-dependencies without explanation

  • crates/cli/Cargo.toml diff — wiremock is removed from [dev-dependencies]. The PR description doesn't mention why.
  • This is fine if no existing tests in other modules depended on it; confirm no compile errors remain in the other test modules (beacon.rs, peers.rs, etc.).

[Low] create_mev_block bypasses epoch boundary refresh

  • mev.rs:387proposer_duties are fetched once per epoch. Inside create_mev_block, when MevError::StatusCodeNot200 triggers a slot advance (next_slot += 1, line 723), the epoch may roll over, but proposer_duties are not refreshed for the new epoch.
  • The fallback at line 681 (fetch_proposers_for_epoch) handles the case where the pubkey is not found, so this won't hard-fail — but it will make an extra network call per epoch boundary that crosses inside the retry loop.

[Low] Hardcoded signature in build_blinded_block_payload is intentional but undocumented

  • mev.rs:765 — The comment-less hardcoded sig_hex is a dummy BLS signature used so the relay returns 400 Bad Request (which request_rtt at line 755 treats as the expected status). This is correct behavior but should have a comment explaining why a dummy/invalid signature is intentional, to avoid future "cleanup" removing it.

Parity Matrix

Component Go Rust Match Notes
CLI flag --endpoints present present yes
CLI flag --beacon-node-endpoint present present yes
CLI flag --load-test present present yes
CLI flag --number-of-payloads present present yes
CLI flag --x-timeout-ms present present yes
Ping test present present yes
PingMeasure test present present partial credential stripping missing in Rust
CreateBlock test present present yes
format_mev_relay_name present present yes
Timeout/cancellation per test emits N failures emits 1 failure no remaining tests silently dropped
Client reuse shared client new per request no performance gap

Tests

Unit tests added:

  • test_format_mev_relay_name — covers standard, no-scheme, no-credential, and short-hash cases. Good coverage.
  • test_get_validator_pk_for_slot — covers found, adjacent, and not-found cases. Good coverage.

No integration tests (requiring live relay) — acceptable given the external dependency.

Clippy/fmt could not be verified (build tools not available in this environment).


Open Questions

  1. Was wiremock removed because beacon.rs/peers.rs tests that used it are also being refactored in a follow-up, or were those tests already removed?
  2. The request_rtt function (mod.rs:691) does not strip URL credentials — is this a known limitation of the shared helper, or should credential handling be added there?

- Follow the `test validator` approach (enum + impl block)
- Simplify error handling for timeouts
- Test is already cancelled by caller appropriately
- Workaround for false positive
Copy link
Copy Markdown
Collaborator

@emlautarom1 emlautarom1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took the liberty of pushing some changes related to CancellationToken, timeouts and the like, that could cause trouble. I did not add them as comments due to the extent of the required changes.

There are a couple of things that require attention, like reusing existing error variants, mismatched features against v1.7.1 and missing tests.

Comment on lines +76 to +81
#[arg(
long = "x-timeout-ms",
default_value = "1000",
help = "X-Timeout-Ms header flag for each request in milliseconds, used by MEVs to compute maximum delay for reply."
)]
pub x_timeout_ms: u32,
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This flag does not exist in v1.7.1 (reference Charon impl). Please remove it.

Comment on lines +623 to +632
r = apply_basic_auth(client.get(&clean_url), creds)
.header("X-Timeout-Ms", x_timeout_ms.to_string())
.header(
"Date-Milliseconds",
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap_or_default()
.as_millis()
.to_string(),
)
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for any of these headers (v 1.7.1)

Comment on lines +644 to +650
let body = resp
.bytes()
.await
.map_err(|e| MevError::Cli(CliError::Other(format!("http response body: {e}"))))?;

let bid: BuilderBidResponse = serde_json::from_slice(&body)
.map_err(|e| MevError::Cli(CliError::Other(format!("http response json: {e}"))))?;
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use existing CliError variants.

Comment on lines +127 to +129
if queued_tests.is_empty() {
return Err(CliError::Other("test case not supported".to_string()));
}
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the _TestCaseNotSupported variant (remove the leading _)

verdict: TestVerdict::Fail,
error: TestResultError::from_string("timeout/interrupted"),
..TestResult::new("")
});
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the CliError::_TimeoutInterrupted variant (remove the leading _)

Comment on lines +298 to +302
};

if resp.status().as_u16() > 399 {
return test_res.fail(CliError::Other(http_status_error(resp.status())));
}
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no support for credentials in v1.7.1.
Note that there are multiple instances of this pattern.

timeout_token.cancel();
});

let start_time = Instant::now();
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use tokio's time module to facilitate mocking time.

Suggested change
let start_time = Instant::now();
let start_time = tokio::time::Instant::now();

Note that there are multiple instances of this pattern.

Comment on lines +544 to +565
) -> Result<BeaconBlockMessage> {
let url = format!("{endpoint}/eth/v2/beacon/blocks/head");
let (clean_url, creds) = parse_endpoint_credentials(&url)?;
let client = reqwest::Client::new();

let resp = tokio::select! {
_ = token.cancelled() => return Err(CliError::Other("timeout/interrupted".to_string())),
r = apply_basic_auth(client.get(&clean_url), creds).send() => {
r.map_err(|e| CliError::Other(format!("http request do: {e}")))?
}
};

let body = resp
.bytes()
.await
.map_err(|e| CliError::Other(format!("http response body: {e}")))?;

let block: BeaconBlock = serde_json::from_slice(&body)
.map_err(|e| CliError::Other(format!("http response json: {e}")))?;

Ok(block.data.message)
}
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • No need to deal with credentials
  • The token is actually not needed, the callers will interrupt this call.
  • We already have CliError variants for these possible errors:
Suggested change
) -> Result<BeaconBlockMessage> {
let url = format!("{endpoint}/eth/v2/beacon/blocks/head");
let (clean_url, creds) = parse_endpoint_credentials(&url)?;
let client = reqwest::Client::new();
let resp = tokio::select! {
_ = token.cancelled() => return Err(CliError::Other("timeout/interrupted".to_string())),
r = apply_basic_auth(client.get(&clean_url), creds).send() => {
r.map_err(|e| CliError::Other(format!("http request do: {e}")))?
}
};
let body = resp
.bytes()
.await
.map_err(|e| CliError::Other(format!("http response body: {e}")))?;
let block: BeaconBlock = serde_json::from_slice(&body)
.map_err(|e| CliError::Other(format!("http response json: {e}")))?;
Ok(block.data.message)
}
async fn latest_beacon_block(endpoint: &str) -> Result<BeaconBlockMessage> {
let client = reqwest::Client::new();
let url = format!("{endpoint}/eth/v2/beacon/blocks/head");
let response = client.get(&url).send().await?;
let body = response.bytes().await?;
let block: BeaconBlock = serde_json::from_slice(&body)?;
Ok(block.data.message)
}

.map(|p| p.pubkey.clone())
}

async fn get_block_header(
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar comments to latest_beacon_block

}

#[cfg(test)]
mod tests {
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are multiple tests missing from testmev_internal_test.go. Let us known if you're going to include them in this PR or you'll add them in a follow up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement cli test mev

3 participants