feat(operations): add release soak test - #176
Open
elyasmnvidian wants to merge 1 commit into
Open
Conversation
|
WalkthroughChangesSoak Test
Estimated code review effort: 4 (Complex) | ~60 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (4)
scripts/soak_test.py (3)
132-149: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win
process_sampleblocks the event loop.It's called from
reporter(Line 442) on the running loop, so thepsfork/exec stalls all in-flight workers and skews the latency samples they're recording. Cheap to make non-blocking.As per coding guidelines: "Use async-first Python APIs".
♻️ Proposed fix
- rss_mib, cpu_percent = process_sample(server_pid) + rss_mib, cpu_percent = await asyncio.to_thread(process_sample, server_pid)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/soak_test.py` around lines 132 - 149, Make process_sample asynchronous and replace the blocking subprocess.run call with an async-first subprocess API, awaiting completion while preserving the existing RSS/CPU parsing and None-return behavior. Update reporter to await process_sample wherever it samples process metrics.Source: Coding guidelines
247-261: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAdd docstrings to the public writer methods.
write_interval,write_error, andcloseare public API entry points; note theMAX_ERROR_RECORDSdrop invariant inwrite_error.As per coding guidelines: "Add concise triple-quoted docstrings for public functions, classes, methods, and API entry points, documenting behavior, important invariants, and relevant errors."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/soak_test.py` around lines 247 - 261, Add concise triple-quoted docstrings to the public methods write_interval, write_error, and close in the writer class, documenting their behavior; explicitly note that write_error drops records after MAX_ERROR_RECORDS and tracks the dropped count.Source: Coding guidelines
876-903: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winTask failure discards
summary.jsonfor the whole run.Raising here exits before the summary block, so a single failed worker after 47 hours leaves only
intervals.csv/errors.jsonland a stderr line. Consider recording the failure into the summary (completed_durationis already false, so it will be marked FAIL) and returning 1/2 instead of skipping artifact generation.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/soak_test.py` around lines 876 - 903, Update the task-result handling loop around run_tasks and worker_results/background_results so failed tasks are recorded for the run and do not raise before the build_summary and summary.json artifact generation. Preserve the failure details in the summary’s failure_reasons, ensure the resulting summary is marked failed, and return the appropriate nonzero status after writing the summary.tests/test_soak_test.py (1)
96-147: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winByte-matching the serialized body is brittle; prefer
respxper guidelines.
request.read().find(b'"stream":true')depends on httpx's compact JSON separators; a serializer change silently flips these tests to the wrong branch without failing. Decode the body instead. Separately, these handlers are the caserespxis meant for.As per coding guidelines: "Use
respxfor HTTP mocking".♻️ Sturdier branch condition
def handler(request: httpx.Request) -> httpx.Response: - if request.read().find(b'"stream":true') >= 0: + if json.loads(request.read())["stream"]:🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_soak_test.py` around lines 96 - 147, Replace the byte-level request-body checks in the handlers for test_send_request_accepts_json_and_streaming_success and test_send_request_rejects_missing_fields_and_non_sse_stream with decoded JSON inspection of the stream field, and migrate these HTTP mocks to respx as required by the project guidelines. Preserve the existing JSON-success, SSE-success, invalid-response, and invalid-stream outcomes for stream=False and stream=True.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/operations/soak_test.md`:
- Line 44: Update the health readiness instruction in the soak test
documentation to require both HTTP 200 and a JSON response containing
{"status":"ok"} from GET /health. Make clear that both conditions must be
satisfied before proceeding.
- Around line 45-46: Update the soak-test instructions to select and pass the
model id returned in each `/v1/models` response, rather than a release route
identifier. Apply this correction to all referenced command examples and
preserve the existing `--model` usage.
- Around line 97-110: Update the pass-criteria checklist in the soak test
documentation to replace the “no inference request fails” requirement with a
requirement that the inference error rate is at or below --max-error-rate, whose
default is 0. Keep the remaining criteria unchanged and align the wording with
the runner’s configured error-budget gate.
In `@scripts/soak_test.py`:
- Around line 403-407: Update the health-check logic around the response JSON
parsing so non-dict JSON bodies are treated as unhealthy rather than calling
.get() on them. Validate that the parsed body is a mapping before reading its
"status" field, while preserving the existing healthy condition and failure
handling in the surrounding try/except.
In `@tests/test_soak_test.py`:
- Around line 179-204: Increase the live soak duration configured in the test
invocation around soak_test.run, and adjust the report and invalid-canary
intervals if needed to preserve multiple metric and canary checks. Keep the
existing success, endpoint, and invalid_request_canaries assertions unchanged
while providing sufficient timing headroom for slower CI runners.
---
Nitpick comments:
In `@scripts/soak_test.py`:
- Around line 132-149: Make process_sample asynchronous and replace the blocking
subprocess.run call with an async-first subprocess API, awaiting completion
while preserving the existing RSS/CPU parsing and None-return behavior. Update
reporter to await process_sample wherever it samples process metrics.
- Around line 247-261: Add concise triple-quoted docstrings to the public
methods write_interval, write_error, and close in the writer class, documenting
their behavior; explicitly note that write_error drops records after
MAX_ERROR_RECORDS and tracks the dropped count.
- Around line 876-903: Update the task-result handling loop around run_tasks and
worker_results/background_results so failed tasks are recorded for the run and
do not raise before the build_summary and summary.json artifact generation.
Preserve the failure details in the summary’s failure_reasons, ensure the
resulting summary is marked failed, and return the appropriate nonzero status
after writing the summary.
In `@tests/test_soak_test.py`:
- Around line 96-147: Replace the byte-level request-body checks in the handlers
for test_send_request_accepts_json_and_streaming_success and
test_send_request_rejects_missing_fields_and_non_sse_stream with decoded JSON
inspection of the stream field, and migrate these HTTP mocks to respx as
required by the project guidelines. Preserve the existing JSON-success,
SSE-success, invalid-response, and invalid-stream outcomes for stream=False and
stream=True.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: c51ae66a-bea9-400b-b5b3-fb971f696e24
📒 Files selected for processing (5)
.gitignoredocs/operations/soak_test.mdmkdocs.ymlscripts/soak_test.pytests/test_soak_test.py
elyasmnvidian
force-pushed
the
emehtabuddin/soak-test
branch
3 times, most recently
from
July 30, 2026 17:16
1dbec57 to
73fab95
Compare
Signed-off-by: Elyas Mehtabuddin <emehtabuddin@nvidia.com>
elyasmnvidian
force-pushed
the
emehtabuddin/soak-test
branch
from
July 31, 2026 05:53
73fab95 to
1ddc528
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a release soak tester as a Rust crate,
crates/switchyard-soak. It sends sustained,closed-loop traffic through a live Switchyard server long enough to surface failures that short
tests miss, and exits non-zero when a release gate fails.
Each run drives:
/v1/chat/completions), Anthropic Messages (/v1/messages), andOpenAI Responses (
/v1/responses)/health,/metrics, and (for a local server) process RSS/CPUIt writes
config.json,intervals.csv,errors.jsonl, andsummary.jsonto a timestampedresults directory, and the summary records the pass result and any failed release gates (error
rate, liveness, metrics, process, canary, counter resets, RSS growth). At the end of a run it also
prints an oha-style latency percentile table (p50 through p99.9) and an ASCII latency histogram to
the terminal.
For monitoring a run in progress, each interval prints a progress line
(
progress=120s/300s(40%) reqs=... errors=...(0.00%) ... status=OK) and overwrites astatus.jsonlive snapshot atomically, with a
statustoken ofok/degraded/stalled— so a remote tail ofthe log or a
cat status.jsongives an at-a-glance confidence check.Run it with:
cargo build --release -p switchyard-soak ./target/release/switchyard-soak --base-url http://127.0.0.1:4000 --model RELEASE_MODEL_ID \ --duration 48h --concurrency 16 --server-pid "$SERVER_PID" --max-rss-growth-mib 512Why
A soak run is the pre-freeze check for changes to libsy, the Rust server, routing, streaming,
translation, or server lifecycle. Writing it in Rust keeps the release tooling on one toolchain and
shrinks the Python surface.
How tested
cargo test -p switchyard-soak— 10 unit tests (duration parsing, percentiles, metric parsing,request bodies, summary gates, timestamps) and 8 integration tests that drive the runner against a
hermetic axum mock (full short run, preflight selection/rejection, streaming/non-streaming success,
invalid-response/stream detection, HTTP-error and empty-stream handling, the canary).
switchyard-serverpointed at the NVIDIA gateway (haiku): a shortrun across all three endpoints (streaming and non-streaming) completed with zero errors, and the
liveness, metrics, process, and canary checks all passed.
switchyard-serverbacked byVidaiMock (a mock LLM server): a clean run passes with zero
errors, and a degraded backend (mock latency past the request timeout) trips the error-rate gate to
a 100% timeout failure and exits non-zero.
scripts/soak-rehearsal.shwires this stack together.cargo clippy -p switchyard-soak --all-targetsandcargo fmtare clean.Docs
docs/operations/soak_test.mdcovers preparing the server, building the tester, rehearsing againsta hermetic VidaiMock backend (
scripts/soak-rehearsal.sh), an optional vegeta throughput probe, the48-hour run, the pass criteria, and how to read the result files.
Checklist
Notes for reviewers
The crate has no dependency on any internal Switchyard crate, so it builds in isolation from the
components/core refactor on main. This branch is not yet rebased onto latest
main; the change isadditive (one new crate plus a workspace-member line, a
.gitignoreentry, and a docs-nav entry),so the rebase is mechanical when wanted.