ratemyagent

Test AI agents like production services.

Agent evaluation usually asks whether an agent can accomplish a task. RateMyAgent asks whether it stays reliable when operated like a production service — under load, latency, faults, and dependency failures.

Think k6 + Chaos Monkey + pytest, but for agents and MCP tools.

---

The problem

Agent reliability is an active area of work: there are task-success benchmarks, adversarial suites, and a growing literature on fault injection for ML and agent systems. The gap this tool addresses is narrower and more practical.

Existing agent evaluation and observability tools generally do not provide an SRE-oriented workflow for systematically injecting operational failures and measuring recovery behaviour.

The widely used tools answer adjacent questions. Langfuse and LangSmith observe production. DeepEval and RAGAS check output quality. MCP-Scan checks whether a tool is malicious. k6 load-tests HTTP endpoints without modelling what an agent does with the failures.

The operational question sits between them:

What happens when your agent's tools and dependencies fail?

That question has a specific shape for agents that it does not have for a web service. An agent retries on its own. It fans out three tool calls in a turn and inherits the p95 of each. It sends malformed arguments as normal traffic, because a model that has just been told a tool exists guesses at its schema. And when a call times out after the work already completed, the retry runs the mutation twice.

RateMyAgent breaks your target on purpose and measures what it does next.

Install

Published on PyPI. Python 3.10+.

pip install ratemyagent

That gives you the engine, the mock targets, and every output format — enough to run a full scan without installing anything else. The adapters that talk to real systems need their SDKs, which are optional so you only pull what you use:

pip install 'ratemyagent[mcp]'          # scan MCP servers over stdio or SSE (mcp SDK 1.x or 2.x)
pip install 'ratemyagent[anthropic]'    # scan Anthropic chat completions
pip install 'ratemyagent[openai]'       # scan OpenAI chat completions
pip install 'ratemyagent[all]'          # all of the above

Prefer uv:

uv tool install ratemyagent             # as a standalone CLI
uv pip install 'ratemyagent[all]'       # into the current environment

From source

For contributors, or to run against an unreleased change:

git clone https://github.com/SMWundefined/RateMyAgent.git
cd RateMyAgent

uv venv --python 3.12
uv pip install -e '.[dev]'              # editable, with pytest and ruff

uv run pytest                           # 665 tests, ~1s, no network or API keys

See Contributing before opening a PR.

30 seconds, no API key

There is a built-in mock target, so you can see the whole thing work before pointing it at anything real. No key, no server, no network.

ratemyagent scan --target mock --profile degraded --requests 40 \
    --concurrency 16 --fault-rate 0.3

RateMyAgent Scan Results
========================

Target: degraded-mock (mock)
Probes: 6/6 complete   Duration: 0.01s

Phase 1  baseline
  Latency ................ p50 3.36s, p95 7.99s, p99 8.48s over 40 requests (0.0% errors)
  Cost ................... 647 in / 120 out tokens per request, no price known for this model
  Concurrency ............ no saturation up to 16 concurrent, sustained 16
  Contract ............... 18 edge cases across 3 tools: 0 rejected cleanly, 18 accepted, 0 crashed

Phase 2  chaos (fault injection)
  Fault tolerance ........ 20 faults injected, 10/10 operations recovered (100%), 1.30x call amplification

Phase 3  behavior analysis
  Behavior ............... 10/10 disrupted operations recovered (100%), 1.30x call amplification, 0 duplicate mutations

                             actual     target     status
  p95 latency                7.99s      5.00s      FAIL
  schema violations accepted 9          0          FAIL
  p99 latency                8.48s      10.00s     pass
  error rate                 0.0%       5.0%       pass
  sustained concurrency      16         5          pass
  contract crash rate        0.0%       0.0%       pass
  recovery rate              100.0%     90.0%      pass
  retry amplification        1.30x      2.00x      pass
  duplicate mutations        0          0          pass
  cost per request           -          $0.1000    n/a

  Score breakdown:
    latency         16/20     (p95 latency was 7,988ms, policy allows at most 5,000ms)
    cost            -/15      (not measured against this target)
    concurrency     15/15
    contract        8/15      (invalid inputs accepted was 9, policy allows at most 0)
    behavior        35/35

  Score: 86/100  (policy production-default)

Latency findings:
  - p95 7.99s and 0.0% errors across 40 requests, with no
    heavy tail, no unusual call overhead, and no error pattern
    to report. Note that zero failures in 40 requests only
    bounds the error rate at roughly 8% (95% confidence), not
    0%. Raise --requests to tighten it.

Cost findings:
  - No published price for model unknown, so token counts are
    reported without a dollar projection. Pass --price-in and
    --price-out to project cost yourself rather than have one
    guessed.

Concurrency findings:
  - No saturation found up to 16 concurrent requests, the
    configured ceiling. The real limit is above 16, so this is
    a floor set by the test, not a measurement of the target
    -- raise --concurrency to find the actual limit.
  - Peak goodput is 4.4 successful req/s at 16 concurrent.
    Past that, added concurrency buys latency and errors
    rather than completed work.

Contract findings:
  - CRITICAL 9 inputs the schema forbids were accepted with a
    success response: missing_required, null_required,
    wrong_type. The tool is not validating what it declares,
    so invalid data reaches whatever it writes to.

Fault tolerance findings:
  - Injected 20 faults across 93 calls (22%): 6 server_error,
    5 rate_limit, 4 connection_refused, 3 timeout, 2
    malformed.
  - Every one of the 10 disrupted operations recovered within
    2 retries.
  - Under fault the latency probe saw a 20% error rate, p95
    8.03s.

Behavior findings:
  - Every one of the 10 disrupted operations recovered.

9 findings across 6 probes. Run with --output agents-md to generate a fix guide.

FAIL: score 86 meets pass threshold 75, but 2 checks failed: p95 latency, schema violations accepted.
Biggest gaps: contract (8/15), latency (16/20).

ratemyagent v0.1.6 - pip install ratemyagent - github.com/SMWundefined/RateMyAgent

Actual sits next to target so the gap is the information. n/a means the probe could not measure this target — those are excluded from the score rather than counted as failures.

The target above is a built-in mock. Before reading a behavior row on a real MCP server the same way, see Known limitations: recovery rate and retry amplification measure the scanner's retry loop, because a server does not retry.

Then point it at something real:

# An MCP server over stdio or SSE
ratemyagent scan --target mcp --uri stdio://./server.py
ratemyagent scan --target mcp --uri sse://localhost:8080/sse --requests 100

# A chat completions endpoint (this one spends money — keep --requests low)
ratemyagent scan --target llm --provider anthropic --model claude-opus-5 --requests 5
ratemyagent scan --target llm --provider openai --model gpt-4o-mini --requests 5

Probing invokes a discovered tool for real, once per request. Pass --tool and --tool-args to choose which one; the default is the first tool the server reports.

Pass real arguments. Without --tool-args, arguments are synthesized from the tool's JSON Schema — correct shape and types, but placeholder values ("ratemyagent probe" for an unconstrained string). A tool that expects a real path, URL or package name will reject all of them, and the scan will accurately measure its rejection path rather than its behaviour. mcp-server-git scores 38/100 on synthesized arguments and 100/100 on real ones -- same server, same repository, same command but for the arguments. The scanner warns when it detects this, but the fastest way to avoid it is:

ratemyagent scan --target mcp --uri "stdio://uvx mcp-server-git" \
  --tool git_log --tool-args '{"repo_path": "/path/to/repo"}'

Also note that probing a mutating tool mutates: scanning write_file writes files.

How a scan works

Three phases, in order. Phase 2 needs phase 1 to compare against; phase 3 reads what phase 2 recorded.

Phase 1 — Baseline. Measures the target as it is: latency distribution, token cost and prompt bloat, the concurrency level where it saturates, and whether its tools honour their own JSON Schema. These are the numbers everything else is compared against.

Phase 2 — Fault injection. A FaultProxy wraps the target and injects timeouts, 429s, 500s, malformed responses and refused connections at a configurable rate. Probes cannot tell they are wrapped, so the same probes run against a sabotaged target and any difference is attributable to the faults.

Phase 3 — Behavior analysis. Reads the trajectory of every operation phase 2 disrupted: did it recover, how long did that take, how many calls did one operation cost, did anything succeed twice. This is the part that is not a load test — it measures behaviour under failure, not failure counts.

Whose behaviour depends on the target. Against something that retries — an agent, or a client wrapping a service — the trajectory is the target's. Against a bare server it is not: the retry loop belongs to the scanner, so recovery latency and call amplification describe RateMyAgent rather than the server. What survives that distinction is target survivability. See Known limitations.

Per-probe detail is in docs/PROBES.md, and the contributor-facing walkthrough of how these pieces fit together is in docs/ARCHITECTURE.md.

Scoring

Results are scored 0–100 against a YAML policy. Probes measure; the policy decides.

• Meeting a threshold scores 100 for that check — a threshold is a limit, not a target.
• Missing it decays linearly to 0 at twice the limit, so a near miss and a catastrophe do not score alike.
• A metric the scan could not produce is skipped, not zeroed. Missing evidence is not a failure.

# my-policy.yaml
name: my-service
thresholds:
  p95_latency_ms: 3000
  error_rate_max: 0.02
  recovery_rate_min: 0.95
  retry_amplification_max: 1.5
  duplicate_mutation_max: 0
pass_score: 80

ratemyagent policy                                    # show the shipped defaults
ratemyagent scan --target mock --policy my-policy.yaml

Passing requires both a score above pass_score and no failed check. The composite is a weighted mean, so a single failure can be averaged down to almost nothing: a recovery rate of 85.7% against a 90% floor scores 95.2, dilutes across the other behaviour checks, and costs well under a point. That produced verdicts like PASS: score 99 printed directly above a table with FAIL in it. The score still summarises; it no longer overrules the evidence beneath it.

FAIL: score 99 meets pass threshold 75, but 1 check failed: recovery rate.

Every threshold is optional, and validation is strict — an unknown key is an error listing the valid ones, because a typo that silently stopped scoring something is worse than a crash. Full reference, including the shipped default explained threshold by threshold: docs/POLICY.md.

CI integration

ratemyagent ci --target mcp --uri stdio://./server.py --policy production.yaml
echo $?     # 0 pass, 1 fail, 2 the scan could not run

Exit code 2 matters: a broken scanner is not a failing target, and a gate that cannot tell them apart is not worth having in a pipeline. Failed checks are printed individually, so a red build says which threshold moved rather than that the score dropped.

ci writes nothing and never prompts. Nothing in the tool does — it stays pipeable.

# .github/workflows/reliability.yml
name: reliability

on: [push, pull_request]

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v5
        with:
          python-version: "3.12"

      - run: uv pip install --system '.[mcp]'

      - name: Reliability gate
        run: |
          ratemyagent ci \
            --target mcp --uri stdio://./server.py \
            --policy production.yaml \
            --requests 120 --concurrency 16 --fault-rate 0.25 \
            --json-out scan.json

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: reliability-scan
          path: scan.json

Use enough requests that the numbers mean something. recovery_rate from the default 20 requests is measured over roughly 4 disrupted operations, which is an anecdote rather than a rate.

AGENTS.md

ratemyagent scan --target mcp --uri stdio://./server.py --output agents-md
# AGENTS.md written to AGENTS.md (7 recommendations, 3 critical)

A fix guide for your target. Each finding states what was observed, why it matters in production, the root cause — weighted toward what AI-generated servers actually get wrong — and a copy-pasteable fix naming the tool that failed:

FINDING: 9 schema-forbidden inputs accepted

Your tool declares required fields and types in its JSON Schema but does not enforce them at runtime. This is common in AI-generated MCP servers where the schema is correct but the handler trusts its input. Every field marked "required" needs an explicit check before the handler touches the data, because the calling agent WILL send malformed arguments — that is normal traffic, not an attack.

Suggested fix for tool "search_database":

if "query" not in args or not isinstance(args["query"], str):
    return {"error": "query is required and must be a string"}

Sections are ordered by severity — duplicate mutations and crashes before latency and cost — so the first thing you read is the thing most worth fixing.

Re-scanning the same file reports movement. Real output, from re-running the generator over the failing mock's guide with the degraded mock:

## Since the last scan

- The previous guide was for `failing-mock`, not `degraded-mock` -- the comparisons below are between two different targets.
- Score improved from 32 to 91/100.
- P95 latency improved from 46.44s to 5.21s.
- Error rate improved from 36.7% to 0.0%.
- Sustained concurrency improved from 0 to 5.
- Schema violations regressed from 4 to 9.
- Edge-case crashes improved from 2 to 0.
- Recovery rate improved from 27% to 100%.
- Retry amplification improved from 1.63x to 1.13x.

The first line is the point: comparing two different targets is usually a mistake, so the generator says so rather than presenting the deltas as a like-for-like improvement.

See the real thing without installing: examples/ has output from a scan of the official mcp-server-git, alongside a deliberately broken mock that triggers every finding at once.

Check the crash detection yourself: examples/mcp_server_git_repro.py sends the same malformed payloads the contract probe sends, using only the MCP SDK, and makes a known-good call after each one to prove the session is still alive. It needs pip install mcp and nothing from this project, so you can confirm a reported crash is real without taking our word for it.

python examples/mcp_server_git_repro.py                    # a throwaway git repo
python examples/mcp_server_git_repro.py --repository .     # your own

Markdown report

ratemyagent scan --target mcp --uri stdio://./server.py --output report
ratemyagent scan --target mcp --uri stdio://./server.py --output all

The whole scan organized by phase, with the actual-vs-target table, the score breakdown, per-level concurrency numbers, every finding, and the settings needed to reproduce the run. Example: examples/mcp-server-git.report.md.

What it can do today

• Latency profiler — p50/p95/p99, TTFT, tool call overhead, heavy-tail detection
• Cost analyzer — tokens per request, prompt-bloat detection and what caching it would save, $/request. Prices are never guessed
• Concurrency tester — ramps 1→N, finds the saturation point and the latency knee
• Contract tester — audits tool schemas and sends six edge-case payloads per tool
• Fault injection — five fault kinds at a configurable rate, deterministic per seed
• Behavior analysis — recovery rate and latency, retry amplification, duplicate mutations, stuck loops
• Adapters — MCP over stdio and SSE; Anthropic and OpenAI chat completions; five mock profiles for testing without any of them
• Outputs — terminal scorecard, markdown report, AGENTS.md, JSON export

Every scan reproduces under --seed. 665 tests, none of which need a network or a key.

Probing writes, unless it knows better

Probing calls a tool for real, once per request, and again under fault injection. Against a read-only tool that is a measurement. Against a write tool it is a hundred writes.

So auto-selection only picks a tool it can establish is read-only. It reads readOnlyHint from the server's own tool annotations, falls back to the tool name, and refuses when neither settles it — a tool nothing classifies is not thereby safe.

refusing to auto-select 'create_entities': it declares readOnlyHint=false.

Probing calls the chosen tool once per request, and again under fault
injection. No tool on this server is known to be read-only, so there is
nothing safe to fall back to.

  tools here: create_entities, create_relations, delete_entities, ...

Choose one yourself, and point the scan at something disposable:
  ratemyagent scan ... --tool <name> --allow-mutating

Naming a tool yourself is a decision the scanner will respect, but a tool known to change state still needs --allow-mutating as a second key:

ratemyagent scan --target mcp --uri ... --tool write_file --allow-mutating

Point that at something disposable. Every scan reports which tool it called and with what arguments, in the scorecard header and in the AGENTS.md state block, so a saved result can always be traced back to what produced it.

Known limitations

Two things this version measures less well than the numbers suggest. Both affect scores you can produce today, so they are stated here rather than in a changelog.

Caller-strategy metrics do not apply to a bare MCP server

Retry amplification, backoff shape and recovery latency describe the scanner's own retry loop, not the target's. A server does not retry — the client does. Point RateMyAgent at an MCP server and those three metrics measure RateMyAgent.

This matters because behaviour carries 35 of the 85 available points against an MCP target (cost is n/a, so it leaves the denominator). A meaningful share of the largest dimension therefore has no subject when the target is a server.

What is real in that dimension is target survivability: whether the server keeps answering while faults are injected around it, and whether operations complete. That part holds. The caller-strategy half will be split out and marked inapplicable for server targets in a future release. Until then, read the behaviour score on an MCP target as survivability plus noise, and do not quote retry amplification or recovery latency for a server.

Scores under synthesized arguments are not comparable to scores under --tool-args

Without --tool-args, arguments are synthesized from each tool's JSON Schema: correct shape and types, placeholder values. A tool that wants a real URL, path or package name rejects all of them, and the scan then measures its rejection path rather than its work.

The gap is not marginal. From this project's own re-scan of mcp-server-fetch:

Arguments	Score
synthesized	38/100
--tool-args '{"url": "https://example.com"}'	100/100

Same server, same command, same seed. The difference is entirely in what we sent it.

--tool and --tool-args are the supported path for any number you intend to rely on. A synthesized-argument score is useful for a first look and for comparing a target against itself; it is not a measurement of the server, and it must not be compared against a score produced with real arguments. The scanner warns when it detects that every synthesized call is being rejected, but the warning is a hint, not a guarantee.

Roadmap

• v1.1 — ratemyagent chaos for targeted single-fault scenarios; streaming TTFT for LLM targets
• v2 — sustained outage windows (current faults are independent per attempt, which models transient failure well and outages not at all); timeout-after-completion faults to exercise duplicate mutations properly; AgentTarget wrapping a Python script; historical trending across scans

Deliberately out of scope: web dashboards, continuous monitoring, framework-specific adapters, security scanning, and anything requiring a database.

Contributing

Set up with the source install above, then:

uv run pytest          # 665 tests, ~1s, no network or API keys
uv run ruff check .

Start with docs/ARCHITECTURE.md — it is written for contributors and covers the Target interface, the FaultProxy, the trajectory model, and the policy engine, including the parts that are load-bearing and the reasoning behind them.

House rules, in short:

• Every probe needs tests that run without API keys, a network, or an MCP server. Use the mock targets in tests/conftest.py.
• Probes measure, the policy judges. A probe that emits a verdict is a bug.
• The FaultProxy is the only place faults are injected.
• No interactive prompts. This is an SRE tool; it has to stay pipeable.
• Say what you measured, not what you would like to be true. Findings call out thin evidence rather than letting it pass quietly.

License

MIT