feat: document-throughput load-test tool - #4376
Open
shumkov wants to merge 2 commits into
Open
Conversation
Add a `load-test` bin to rs-scripts for driving and measuring sustained document load against a running network (block-interval / throughput / round-stability stress measurement, not a max-TPS benchmark). It drives DocumentCreate state transitions from a single funded identity, fanned out across N cloned data-contract variants so each contract's in-flight, not-yet-committed nonces stay well under the consensus missing-revisions window (nonce uniqueness itself comes from the SDK's shared per-(identity,contract) nonce cache). Requests are paced by a token interval + semaphore and are fire-and-forget (broadcast to mempool, retries off) so the measured rate reflects real acceptance. `--quorum-http` selects an external-client context provider (TrustedHttpContextProvider) for testnet/mainnet; `--core-*` uses Core RPC for devnets with no HTTP quorum endpoint. Ships a cheap, spammable person-doc-type fixture (docs/load-tool/load-contract.json). The bin takes an already-funded identity (id + private key) and only talks to DAPI; identity bootstrap/funding is intentionally out of scope and will land separately on platform-wallet's asset-lock funding path. Additive only: one new bin in an internal scripts crate, no existing code changed. Exercised end-to-end on testnet (dash-testnet-51, protocol 13) as an external client: sustained ~15 docs/s at round 0 across 13 DAPI nodes, with honest document service unaffected (0.02% rejects) under a concurrent 100-lane consensus-flood. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughAdds a ChangesLoad-test execution
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Operator
participant load_test
participant DAPI_or_Core_RPC
participant Identity
participant Dash_Platform
Operator->>load_test: Provide load-test options
load_test->>DAPI_or_Core_RPC: Build SDK through quorum data
load_test->>Identity: Fetch identity and select signing key
load_test->>Dash_Platform: Register contract variants
load_test->>Dash_Platform: Broadcast generated documents
Dash_Platform-->>load_test: Return submission results
load_test-->>Operator: Report progress and throughput
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
shumkov
changed the title
Collaborator
|
🔍 Review in progress — actively reviewing now (commit 808e1a0) |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (4)
packages/rs-scripts/src/bin/load_test.rs (3)
278-285: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winGive the explicit flag precedence over the environment variable.
std::env::var("CORE_RPC_PASSWORD").ok().or(args.core_password.clone())uses the environment value even when the user passes--core-password. An exported variable then silently overrides an explicit flag. Reverse the order.♻️ Proposed change
- let core_password = std::env::var("CORE_RPC_PASSWORD") - .ok() - .or(args.core_password.clone()) + let core_password = args + .core_password + .clone() + .or_else(|| std::env::var("CORE_RPC_PASSWORD").ok())🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/rs-scripts/src/bin/load_test.rs` around lines 278 - 285, Update the core_password resolution in the load-test argument handling to prefer args.core_password from the explicit --core-password flag before falling back to CORE_RPC_PASSWORD, while preserving the existing missing-password error message and behavior.
543-549: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueReturn a failure signal when every broadcast fails.
run_document_loadreturns(), sorunreturnsOk(())and the process exits with code 0 even when all broadcasts fail. Automated runs cannot detect a total failure. Return the counters and map an all-error run toExitCode::FAILURE.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/rs-scripts/src/bin/load_test.rs` around lines 543 - 549, Update run_document_load to return its success and failure counters after draining the semaphore, then have run inspect those counters and return ExitCode::FAILURE when every broadcast fails; preserve the successful exit behavior when at least one broadcast succeeds.
374-393: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueUse exponential backoff for the registration retry.
The retry uses a fixed 4 s delay for 8 attempts. During a validator restart, this fails after 32 s. Exponential backoff tolerates longer outages with the same attempt count.
Note also that a failure after acceptance (a lost response or a wait timeout) leaves an orphan contract on chain, because the retry consumes the next identity nonce. This does not break the run. Record it as a known behavior in the module docs.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/rs-scripts/src/bin/load_test.rs` around lines 374 - 393, The retry loop around put_to_platform_and_wait_for_response should use an exponential backoff rather than a fixed 4-second delay, increasing the wait between attempts while preserving the existing attempt limit and error handling. Also update the module documentation to record that failures after on-chain acceptance can orphan a contract because retries consume the next identity nonce.packages/rs-scripts/Cargo.toml (1)
40-40: 🗄️ Data Integrity & Integration | 🔵 TrivialKeep
rand = "0.8"and run the required workspace checks.rs-scriptsanddppboth resolve torand 0.8.6; the workspace has no sharedranddependency. Run workspace-widecargo test,cargo check,cargo clippy, andcargo fmt.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/rs-scripts/Cargo.toml` at line 40, Keep the rand dependency at version 0.8 in the rs-scripts Cargo manifest, and run the required workspace-wide cargo test, cargo check, cargo clippy, and cargo fmt checks.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/rs-scripts/src/bin/load_test.rs`:
- Around line 550-562: Update the completion reporting around the load-test loop
to calculate throughput from successfully accepted documents only (`ok /
measured_elapsed_seconds`), using wall-clock elapsed time that includes
semaphore backpressure and draining after the deadline. Change the output label
and target comparison to reflect accepted docs per second, while preserving the
existing success/error counts.
- Around line 112-116: Update the private_key clap argument to support
LOAD_TEST_PRIVATE_KEY via env and hide its environment value by adding the
requested env and hide_env_values settings. Also enable clap’s env feature in
the package dependency configuration while preserving existing CLI argument
behavior.
---
Nitpick comments:
In `@packages/rs-scripts/Cargo.toml`:
- Line 40: Keep the rand dependency at version 0.8 in the rs-scripts Cargo
manifest, and run the required workspace-wide cargo test, cargo check, cargo
clippy, and cargo fmt checks.
In `@packages/rs-scripts/src/bin/load_test.rs`:
- Around line 278-285: Update the core_password resolution in the load-test
argument handling to prefer args.core_password from the explicit --core-password
flag before falling back to CORE_RPC_PASSWORD, while preserving the existing
missing-password error message and behavior.
- Around line 543-549: Update run_document_load to return its success and
failure counters after draining the semaphore, then have run inspect those
counters and return ExitCode::FAILURE when every broadcast fails; preserve the
successful exit behavior when at least one broadcast succeeds.
- Around line 374-393: The retry loop around
put_to_platform_and_wait_for_response should use an exponential backoff rather
than a fixed 4-second delay, increasing the wait between attempts while
preserving the existing attempt limit and error handling. Also update the module
documentation to record that failures after on-chain acceptance can orphan a
contract because retries consume the next identity nonce.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 0cc1713a-78e6-41d5-b9c2-7c90cc70d941
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (3)
docs/load-tool/load-contract.jsonpackages/rs-scripts/Cargo.tomlpackages/rs-scripts/src/bin/load_test.rs
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## v4.2-dev #4376 +/- ##
=========================================
Coverage 87.80% 87.80%
=========================================
Files 2641 2641
Lines 336510 336510
=========================================
Hits 295468 295468
Misses 41042 41042
🚀 New features to boost your workflow:
|
thepastaclaw
left a comment
thepastaclaw
left a comment
Collaborator
There was a problem hiding this comment.
Final validation — Codex/Sol only (Phase 2 disabled)
The new binary is isolated from consensus-critical paths, but it has four measurement/bootstrap correctness issues and one secret-handling issue: deadline waits can submit after the configured window, accepted throughput counts failures, extreme timer inputs can panic or yield a meaningless run, ambiguous registration retries can create paid duplicate contracts, and the private key lacks a non-argv input path. These are suggestions rather than consensus-blocking defects, so the appropriate action is COMMENT.
Source: reviewers codex general (gpt-5.6-sol) and codex rust-quality (gpt-5.6-sol); final verifier codex (gpt-5.6-sol). openclaw-agent/cliproxy/gpt-5.6-sol is orchestration-only and not reviewer evidence.
Validated zero-blocker Codex/Sol precheck evidence was promoted to final because Phase 2 (Sonnet/Opus) is temporarily disabled. This is Codex/Sol-only final validation, not Codex + Sonnet/Opus coverage.
Review provenance
- Codex reviewers:
gpt-5.6-sol— general (completed),gpt-5.6-sol— rust-quality (completed) - Verifier:
gpt-5.6-sol— verifier - Sonnet/Opus: not run (Phase 2 disabled — temporary Codex/Sol-only final)
- Secondary pass: disabled (
temporary_phase2_sonnet_disable)
🟡 5 suggestion(s)
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `packages/rs-scripts/src/bin/load_test.rs`:
- [SUGGESTION] packages/rs-scripts/src/bin/load_test.rs:461-467: Stop deadline is not enforced while waiting for a tick or permit
The deadline is checked only before awaiting the ticker and semaphore. Either wait can finish after the configured duration, after which another document is still submitted. For example, `--rate 0.1 --time 1` submits on the interval's immediate first tick, waits until the ten-second second tick, and submits again even though the run was requested to last one second. Semaphore backpressure can produce the same behavior for up to the request timeout. Race both waits against a sleep ending at `deadline`, and exit without spawning when the deadline wins.
- [SUGGESTION] packages/rs-scripts/src/bin/load_test.rs:377-385: Retrying the combined put-and-wait call can register duplicate contracts
`put_to_platform_and_wait_for_response` first obtains a fresh identity nonce and broadcasts the resulting transition, then separately waits for its confirmation proof. If the broadcast succeeds but confirmation waiting times out or loses its response, this retry loop invokes the whole operation again. The SDK then allocates another nonce, deriving a different contract ID, so several paid contracts can be committed while only the final confirmed one is tracked. Construct and broadcast each contract transition once, retain that transition and contract ID, and retry only confirmation/querying after an ambiguous post-broadcast failure; fresh-transition retries should be limited to errors known to occur before broadcast acceptance.
- [SUGGESTION] packages/rs-scripts/src/bin/load_test.rs:188-190: Timer validation still permits panic and zero-duration cases
Positive infinity and sufficiently large finite rates pass the current check. Their reciprocal becomes a zero `Duration`, and `tokio::time::interval` panics because its period must be nonzero; this occurs only after contract variants have been registered and fees may have been spent. Separately, `--time 0` performs paid contract registration, broadcasts nothing, and later computes `0 / 0`, reporting `NaN docs/s` with a successful exit. Reject non-finite rates, rates whose effective interval is zero, and zero run durations before any network side effects.
- [SUGGESTION] packages/rs-scripts/src/bin/load_test.rs:550-560: Accepted throughput is calculated from all attempts
The summary labels the result as `docs/s accepted`, but its numerator is `ok + err`, so failed broadcasts inflate the tool's primary metric. It also divides by the configured duration even when ticker/semaphore waits and draining extend the actual run. Use successful acceptances as the numerator and measured elapsed wall-clock time as the denominator; the attempted and error counts are already reported separately.
- [SUGGESTION] packages/rs-scripts/src/bin/load_test.rs:112-116: Provide a non-argv input path for the identity private key
The identity private key can currently be supplied only through `--private-key`, exposing it in the process argument list and commonly in shell history. Add support for a dedicated environment variable such as `LOAD_TEST_PRIVATE_KEY` while preserving the CLI option, hide the environment value in clap-generated output, and enable clap's `env` feature in `packages/rs-scripts/Cargo.toml`.
thephez
reviewed
Aug 11, 2026
thephez
left a comment
thephez
left a comment
Collaborator
There was a problem hiding this comment.
This tool serves a different purpose than the strategy tests (https://github.com/dashpay/platform/tree/v4.2-dev/packages/strategy-tests)? I thought it did a similar thing (although I haven't personally used it).
Address PR review on the load-test bin: - Reject non-finite (+inf/NaN) and too-high --rate whose reciprocal rounds to a zero tick period (previously panicked tokio's interval, but only after contract variants were already registered and paid for), and reject --time 0. - Base reported accepted-throughput on successful acceptances over measured wall-clock (ok / start.elapsed()) instead of (ok + err) over the nominal --time, so failed broadcasts and post-deadline drain no longer inflate the tool's headline metric. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue being fixed or feature implemented
Platform had no in-repo tool to generate sustained, honest document-throughput
load against a running network for consensus stress measurement (block
interval / throughput / round stability). This adds one to
rs-scriptsasdev/ops tooling. It was used to produce the consensus-DoS "Gate C" stress
datapoints and the testnet post-roll fair-share flood/load pass.
What was done?
A new
load-testbin inpackages/rs-scripts, alongside the existingregister-contract/check-contract-properties:load-test— drivesDocumentCreatestate transitions from a singlefunded identity, fanned out across N cloned data-contract variants. Nonce
uniqueness comes from the SDK's shared per-
(identity, contract)noncecache; the fan-out keeps each contract's in-flight, not-yet-committed nonces
under the consensus missing-revisions window so fire-and-forget creates are
not rejected for running too far ahead of the committed nonce. Paced by a
token interval + semaphore; fire-and-forget (broadcast to mempool, retries
off) so the measured rate reflects real acceptance. Two quorum-key sources
for proof verification:
--quorum-http(external client;TrustedHttpContextProvider, e.g. testnet/mainnet) or--core-*(devnetswith no HTTP quorum endpoint, via Core RPC).
docs/load-tool/load-contract.json— a cheap, non-unique-index "person"document-type fixture that is freely spammable.
packages/rs-scripts/Cargo.toml— one[[bin]]entry plus the featureflags the bin needs (
dpp:random-documents;tokio:sync,time) andrand.The bin takes an already-funded identity (id + private key) and only talks to
DAPI. Identity bootstrap/funding is intentionally out of scope for this PR and
will land separately on
platform-wallet's proven asset-lock funding path.How Has This Been Tested?
Exercised end-to-end against live networks:
~15 docs/s held at round 0; rounds held 0 up to ~55 tx/s committed;
single-client ceiling ~30 tx/s (DAPI-gateway bound, not chain).
dash-testnet-51, protocol 13), as an external client via--quorum-httpacross 13 HPMN DAPI nodes:round 0, node memory flat.
8,719 ok / 2 err (0.023%) — unaffected; round 0 throughout; tenderdash
memory flat 128–155 MiB; the flooded node recovered identically to a
non-flooded peer.
cargo build -p rs-scripts,cargo clippy --workspace --all-features, andcargo fmt --all --check— all clean.Degenerate-input guards (
--rate 0,--connections 0,--contracts < --connections) verified to fail loud instead of hanging.This is an operational/measurement bin in an internal scripts crate (like the
existing
rs-scriptsbins) exercised against live networks; it carries no unittests.
Breaking Changes
None — additive new bin in an internal scripts crate; no existing code changed.
Checklist:
Summary by CodeRabbit