Compare built-in and Rust load on FYRE

[lucarlig]

The existing load workflow could not produce a fair built-in-dataplane versus external-dataplane comparison. The lanes could use different MCP client behavior, a single Locust process capped load generation, distributed worker failures could be hidden, and the FYRE workflow required manual infrastructure and artifact handling.

This PR releases cf-integration 0.5.0 with one repeatable CLI workflow for local load testing and FYRE comparison campaigns. The default FYRE command provisions the infrastructure, bootstraps it with Ansible, runs the complete comparison, downloads raw evidence, generates reports, and destroys only VMs owned by that run.

Default FYRE comparison

cf-integration load fyre run provisions three Ubuntu 24.04 VMs:

Role	Default allocation	Work
Load generator	4 vCPU / 16 GB	Locust 2.46.2, FastHttpUser, three distributed workers, zero wait
Shared target	4 vCPU / 4 GB	External dataplane and built-in dataplane run sequentially on the same VM
Backend	8 vCPU / 32 GB	Fast Time MCP server, six nonfailure tools, explicit zero delay

The eight default measurements are the built-in dataplane and external dataplane at 125, 250, 500, and 1,000 users. Every point uses a 30-second ramp, a 30-second steady warmup, a statistics reset, and a 60-minute measured window.

Both lanes use the same stateless MCP 2026-07-28 Locust client and logical workload. Each user performs server/discover before measurement; measured traffic contains only one tools/call per request, uniformly selected from convert-time, echo, get-stats, get-system-time, schema-success, and verify-protocol. There are no measured resources/read, prompt, or request fan-out calls. The built-in gateway owns any upstream protocol/session behavior.

The target stack is reset between lanes. The built-in lane uses the Python gateway with four workers, PostgreSQL, and Redis. The external lane uses the Rust dataplane, Redis, and a loopback JWKS helper. The built-in fixture image is pinned to public multi-architecture manifest ghcr.io/contextforge-org/cf-integration-fixture@sha256:5b206e6f863cea9f8cabea6451392428fbe67b16bdf10fefd1f7ac8032d95b52, built from IBM/mcp-context-forge commit 33e2dd93a53a9cc2c5088b731822dfec4852fa2e on the MCP SDK v2 branch.

Commands

# Complete default 4 vCPU / 4 GB comparison
cf-integration load fyre run --run-id builtin-external-hourly
cf-integration l f r -i builtin-external-hourly

# Inspect or clean up a retained run
cf-integration load fyre status --run-id builtin-external-hourly
cf-integration load fyre destroy --run-id builtin-external-hourly
cf-integration l f s -i builtin-external-hourly
cf-integration l f d -i builtin-external-hourly

# Same eight measurements on a 2 vCPU / 2 GB target
cf-integration load fyre run \
  --file benchmarks/fyre/comparison-2v2.yaml \
  --run-id builtin-external-2v2

The earlier Rust-only capacity search remains available through vertical-low-memory.yaml. Comparison report labels and allocations come from the selected profile rather than a hard-coded target size.

Fail-fast, fairness, and artifacts

The campaign smokes all six tools before every measured point and stops immediately after the first request or distributed-worker error. Worker IDs are included in fail-fast diagnostics, and each worker receives a different replica offset so multi-replica traffic remains balanced.

Telemetry records host and container CPU, per-core pressure, memory, swap/OOM state, scheduling pressure, socket counters, worker health, virtualization steal, and per-replica request rates. Sustained helper saturation invalidates the partial campaign. The CLI grows only the saturated helper, archives the invalidated evidence, and reruns all eight points so the final comparison uses one helper allocation.

Artifacts and recovery state are stored below $CF_INTEGRATION_DIR/fyre/<run-id>/:

• results/summary.csv and results/summary.json
• results/slack-comparison.png, including external-dataplane RPS divided by built-in-dataplane RPS
• raw Locust CSV, HTML, JSON, worker logs, and telemetry under results/comparison/<lane>-<users>/
• manifest.json with exact images, workload, inventory, and allocations

Artifacts are downloaded after every phase and before cleanup. Cleanup retries Terraform destroy three times, preserves recovery state on failure, and refuses to touch VMs outside the run-owned Terraform state. FYRE expiry remains the final cleanup backstop.

Infrastructure and security

Terraform owns FYRE provisioning and isolated state; pinned Ansible bootstraps Docker and Compose on each VM. The CLI preflights product-group CPU, memory, standalone-VM disk, and public-IP quota before provisioning.

Benchmark services bind only to FYRE private addresses or loopback; public interfaces are used only for SSH orchestration. Provider credentials stay in environment variables and are excluded from manifests, state summaries, logs, and command arguments. Inventory-derived backend URLs are shell-quoted in every remote command.

The removed global dataplane cache-expiry override is intentional. These Compose overlays are packaged implementation assets owned by the CLI, which now publishes the required global configuration before dataplane startup and uses isolated scenario identities. Direct long-lived consumers of the internal Compose fragments should use the CLI workflow rather than depending on the removed override.

The conformance token/config changes remain in this release because FYRE and standalone external workflows use the same CLI-owned JWKS, token, and Redis snapshot lifecycle. Keeping that lifecycle in the packaged CLI avoids environment-specific auth setup.

Other load improvements

• Adds -w/--workers, -i/--isolate-cpus, and -m/--builtin-memory-limit.
• Splits Docker CPUs between target and Locust when isolation is requested.
• Raises target and Locust file-descriptor limits to 65,536.
• Removes the previous 50–200 ms client think time.
• Uses explicit zero Fast Time delay.
• Provides visible one-letter aliases for every public command and short flags for every public option.

Validation

• cargo test --all-targets --locked — 431 passed
• python3 -m unittest discover -s benchmarks/fyre -p 'test_*.py' — 24 passed and now runs in CI
• cargo clippy --all-targets --locked -- -D warnings
• cargo fmt --all --check
• Comparison JSON/CSV and Slack PNG render exercised with the pinned Matplotlib runtime
• Public pinned SDK v2 image: six-tool modern smoke passed
• Distributed published-image validation: 7,507 calls, 0 failures in the short correctness run

[msureshkumar88]

Review summary

Reviewed against the PR's own stated goals (no linked issue exists in this repo). Local-load fixes are minimal and correct; FYRE campaign is proportionate to the stated need. Requesting changes on 3 items before merge.

Blocking

1. benchmarks/fyre/campaign.py:272-278 — command injection (CWE-78): inventory["fast_time"]["private_ip"] is concatenated directly into a remote shell command string instead of shlex.quote()'d, inconsistent with every other remote command in this file. One-line fix.

2. Undocumented possible breaking change: docker/docker-compose.cf-dataplane.yaml and -standalone.yaml silently drop the CONTEXTFORGE_DATA_PLANE_USER_CONFIG_CACHE_EXPIRY_SECONDS override (previously commented as needed to prevent stale-config visibility under steady traffic). The new per-scenario Redis global_config_writer path only fixes this for conformance runs with a fresh UUID subject per scenario — any other consumer of these compose files driving them directly with a long-lived stack loses the old protection with no flag/opt-out/changelog note. Please confirm this is intentional and either restore an equivalent short/zero-TTL default for the general path, or call it out explicitly as a breaking change with a migration note.

3. Scope creep: src/runtime/conformance/mod.rs, src/helpers/config.rs/mod.rs, and the new docker/docker-compose.cf-dataplane-config.yaml implement an unrelated conformance token-issuance rework (per-scenario locally-minted tokens replacing one shared revocable catalog token) plus a new Redis config-publish feature. Neither implements any of the four problems this PR states (Locust single-process cap, fd limits, worker-exit-code propagation, VM-comparison capability). Recommend splitting into its own PR for a clean review/revert trail, ideally reviewed by a conformance/auth owner.

Should-fix (non-blocking)

• benchmarks/fyre/terraform/main.tf — no firewall/security-group resource restricts inbound to the provisioned VMs; dataplane:4445 and fast-time:9080 (host network) are reachable from the public internet, and Fast Time's /health has no auth.
• src/runtime/fyre.rs — the async orchestration paths (helper-saturation/archive/restart in run_fyre_matrix, terraform_destroy's retry loop) have zero test coverage; only pure/sync config-validation functions are tested. Highest blast-radius untested code in the PR given it's credential-handling + billed-cloud-spend logic.
• benchmarks/fyre/report.py and deploy/monitor.py (producer side) — zero test coverage; the numbers that land in the published comparison report/PNG are unverified.
• src/runtime/stack/mod.rs — --isolate-cpus/--builtin-memory-limit are only proven at CLI-parse/static-YAML level, not proven end-to-end to populate the env vars on the executed command.
• src/runtime/fyre.rs:938 vs src/app.rs — validate_run_id duplicated verbatim in both files; this is the path-traversal guard and needs to stay in sync by hand.
• scripts/locustfile_mcp.py — _TARGET_SEQUENCE round-robin counter for MCP_BASE_URLS is per-process, not shared, so under --workers > 1 traffic skews toward low-index replicas — undercuts the exact multi-replica comparison FYRE is built for.
• scripts/locustfile_mcp.py — master-side "distributed worker failed" log omits the worker/node id, can't attribute which worker failed with multiple workers running.
• PR description says "FYRE campaign unit tests — 9 passed"; actual test_campaign.py has 16 tests (verified by running the suite). Please correct the PR body.
• release: prepare cf-integration 0.5.0 commit is buried mid-branch with 12 more commits appending to the same changelog section afterward — risks a premature/incomplete tag given release-plz.toml has tag/release automation enabled. Consider moving the version bump to the tip.

Everything else checked out clean: no dead code, no hardcoded secrets, no yaml.load/shell=True patterns, credential handling in fyre.rs verified sound (never in args/logs/manifests/state), fd-limit/cpuset changes empirically verified safe when the new flags aren't used, docs match implementation elsewhere. No linked GitHub issue exists in this repo — recommend filing one for traceability.

[lucarlig]

Addressed the review on the latest branch:

• Fixed the remote-shell injection finding by quoting the inventory-derived Fast Time URL before composing both registration commands.
• Bound target and Fast Time services to FYRE private addresses, and moved the distributed Locust coordinator/workers onto a run-scoped Docker network.
• Added per-worker replica offsets and worker IDs in fail-fast diagnostics.
• Added comparison coverage for the eight-run sequence, first-error stop, helper-saturation restart request, and container startup; the FYRE suite now has 21 tests. Updated the PR body and check counts.

The removed global cache override remains intentional. These Compose files are internal assets owned by this development CLI; direct long-lived Compose consumers are not a compatibility target. The CLI config writer and isolated per-scenario identities are the supported path, so I did not add a compatibility shim.

I also kept the conformance token/config work in this PR because it fixes the CLI-owned JWKS/config lifecycle used by these workflows, and the requested scope is to keep the complete dev-tool change in PR #17 rather than split it.

[lucarlig]

@msureshkumar88 I rechecked each review item against the current branch and pushed the remaining fixes.

Blocking items

1. Remote-shell quoting: fixed. The newer comparison path was quoted, but the older scaling prepare_hosts path still had one direct interpolation. Commit 44923a8 now quotes the complete inventory-derived backend URL and protocol argument and adds an injection regression test.
2. Cache-expiry removal: intentional; I did not restore the compatibility override. These Compose fragments are CLI-owned packaged assets. The CLI now publishes global config before dataplane startup and uses isolated scenario identities. The PR description now states the contract and migration path explicitly.
3. Conformance/config scope: intentionally retained. FYRE and standalone external runs share the CLI-owned JWKS, token, and Redis snapshot lifecycle, so splitting it would leave this release dependent on environment-specific auth setup.

Other findings

• Benchmark services now bind only to FYRE private addresses or loopback; public interfaces have no listeners for ports 4444, 4445, or 9080. SSH remains the only public-interface use.
• FYRE coverage is now 24 tests and runs in CI. It covers the eight-run comparison, first-error stop, helper-saturation restart request, worker lifecycle, measurement-window filtering, per-replica statistics, shell quoting, comparison CSV/JSON generation, and monitor CPU/memory calculations. A full comparison PNG render also passes with the pinned Matplotlib runtime.
• Replica selection now uses a worker-specific offset, and distributed fail-fast messages include the worker ID.
• The duplicate run-ID validator was removed; the current branch has one implementation in runtime/fyre.rs.
• The 0.5.0 release notes are finalized at the branch tip in 2853135, including the later 2-vCPU profile and report-label fixes.
• The local load flags flow through the resolved load request into the executed Compose environment, with Compose contract tests for both CPU-set variables and the gateway memory limit. I did not add a synthetic Terraform-provider retry test; cleanup ownership/refusal is unit-tested, while the three-attempt destroy path was exercised by the live FYRE runs.

Current validation: 431 Rust tests passed, 24 FYRE tests passed, Clippy is clean with warnings denied, formatting passes, and comparison JSON/CSV/PNG rendering succeeds.

[msureshkumar88]

Re-review

Verified the 3 items from the prior review round directly against the code (not the PR comments):

• Command injection (CWE-78) in campaign.py — fixed. shlex.quote() now wraps backend_url in both the older scaling path and the new rust/builtin comparison paths (lines 301-302, 1045, 1102). Regression test injects "10.0.0.2; touch /tmp/unquoted" as private_ip and asserts the quoted form.
• Cache-expiry removal disclosed as intentional breaking change — done, PR body documents it.
• Conformance/config scope creep — not split out; reasoned justification given (shared JWKS/token/Redis lifecycle). Judgment call, acceptable.

Also verified: FYRE-private-only bind IPs, per-worker MCP_REPLICA_OFFSET plumbing, worker ID in fail-fast diagnostics, duplicate validate_run_id removed, test count claim accurate, release notes at tip.

Fresh review + adversarial verification

Ran a fresh pass and then adversarially verified every finding against the code before trusting any of it (including my own). Net: no blocking defects survived verification. Two initially-flagged items collapsed because the reviewer (me) stopped grepping at the repo boundary — GATEWAY_MEM_LIMIT is consumed by the upstream mcp-context-forge compose file pulled into the controlplane checkout, not by anything in this repo, and the dataplane cache-expiry change is a semantics inversion (TTL=0 disables the cache for functional tests; FYRE's :60 restores the image default for realistic throughput, not a kept-vs-dropped security control).

No alembic migration touched. No linked issue number in the PR body.

Small non-blocking cleanups worth taking

1. benchmarks/fyre/campaign.py:1046 — protocol_version is interpolated into the SSH command unquoted, while the equivalent site at :302 quotes it (that site was hardened in the last review round for the same class of issue). Value is operator-controlled from committed campaign YAML, so this is a consistency/hardening gap rather than a remotely exploitable injection, but it should match its sibling.
2. .env.example:99-101 — CF_DATAPLANE_USER_CONFIG_CACHE_EXPIRY_SECONDS documentation is now dangling; its only consumer (docker-compose.cf-dataplane*.yaml) was removed by this PR.
3. benchmarks/fyre/report.py:41 (rust["rps"] / builtin["rps"]) — theoretically divides by zero if a phase completes with zero requests and zero failures. Current locust timeout/fail-fast config makes this practically unreachable, but guarding it at the source (read_stats: passed = failures == 0 and requests > 0) is cheap and also covers the same pattern at report.py:229,255.
4. src/runtime/fyre.rs — terraform_destroy's retry/backoff and the helper-saturation restart/archive path have no test coverage. Verified the logic itself is currently correct (no off-by-one), so this is a coverage request, not a defect. ProcessRunner injection seam already exists with fakes in-tree, so a #[tokio::test] is cheap to add.

None of these block merge. Approving.

🤖 Generated with Claude Code