From 135b0653169dddb80e37cab50b676cd715466980 Mon Sep 17 00:00:00 2001 From: Luke Parke <5702154+LukasParke@users.noreply.github.com> Date: Thu, 5 Nov 2026 16:16:04 -0600 Subject: [PATCH] docs: rewrite the README for a public audience, add CONTRIBUTING and SECURITY The README now explains what the benchmark measures (the four checks, reward, outcome_code), how determinism works and what the results so far are, and drops internal deployment notes. The arms note cites the 1,595 MCP tools the snapshots hold. Co-Authored-By: Claude Opus 5.5 --- CONTRIBUTING.md | 27 ++++++++++ README.md | 84 +++++++++++++++++++++++++------- SECURITY.md | 7 +++ docs/research/arms-2026-10-31.md | 2 +- 4 files changed, 101 insertions(+), 19 deletions(-) create mode 100644 CONTRIBUTING.md create mode 100644 SECURITY.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..f769bf61 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,27 @@ +# Contributing + +Start with `docs/authoring.md`. It covers what a mock service must implement, how to write a +scenario (instruction, dataset, expectations, oracle solution) and the realism and determinism +rules both must follow. + +## Before opening a pull request + +```bash +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test --workspace +cargo run -q -p scenario --bin taskgen # validates every scenario, regenerates tasks/ +``` + +- Commit the regenerated `tasks/` together with the change that caused it. CI runs + `taskgen --check` and fails when `tasks/` is stale. Image tags are content hashes, so any + change under `crates/`, `scenarios/`, `servers/`, the catalog or the image Dockerfiles changes + them. +- For a new or changed scenario, run `deploy/oracle.sh ` (needs Harbor and Docker): + the oracle agent must score 1 and the nop agent 0. +- Don't edit files under `tasks/` by hand. + +## Data + +Datasets are invented. Never add real customer, account or personal data, and never add a real +credential: tokens in the catalog and datasets are placeholders the mocks accept. diff --git a/README.md b/README.md index 3476afb1..b513e0c3 100644 --- a/README.md +++ b/README.md @@ -1,42 +1,81 @@ # tool-bench -A deterministic benchmark for tool calling with very large tool sets: 30 real services, about -1,500 MCP tools, and more than 600k tokens of tool definitions. Agents run through Harbor on the -OpenRouter benchmark flow and call real (pinned) MCP servers. Those servers talk to mock APIs -written in Rust, which answer like the real services from curated per-scenario datasets. +A deterministic benchmark for tool calling with very large tool sets. Each trial attaches 30 +services through 32 MCP endpoints, 1,595 tools and about 727k tokens of tool definitions, then +asks the agent to do one realistic task: file an issue, scale a deployment, refund an order +across Zendesk and Stripe. -Design: `docs/superpowers/specs/2026-10-30-tool-bench-design.md`. Operating it: `docs/runbook.md`. +The MCP servers are the real, pinned upstream servers (GitHub, Kubernetes, AWS, Stripe, …). +They talk to mock APIs written in Rust that answer like the real services from a curated +dataset per scenario. Nothing leaves the trial except model traffic, and every run of a scenario +starts from the same world. -## Status +## What it measures -- **30 services, 32 MCP endpoints, 1,595 tools, ~727k o200k tokens** of tool definitions (`docs/research/tool-payload.md`), captured offline and hash-identical across runs. -- **150 scenarios** (120 single-service, 30 cross-service), each a generated Harbor task in `tasks/`. Every one scores reward 1 with Harbor's oracle agent (replaying the solution over MCP) and 0 with the nop agent. -- **Determinism:** logical clock, seeded IDs, recorded journal; `world-replay` re-issues a trial's journal against a fresh world and must match byte for byte. -- **Model smoke run:** 3 tasks × 2 modes with Claude Code + Claude Haiku 5.5 via Ori/OpenRouter, all reward 1.0 (`docs/research/smoke-2026-10-31.md`, `deploy/smoke-ori.sh`). Full payload ≈ 820k Claude tokens per turn. -- **Deferred-tools arms:** universal vs native ToolSearch vs all-tools-upfront, 324 runs over Claude Haiku 5.5 and Kimi K2.7 Code (`docs/research/arms-2026-10-31.md`, `deploy/smoke-arms.sh`). -- **Argo:** images are published on merge to `main` (`docs/runbook.md`); the Harbor runner registers this directory's `tasks/`. -- **Provenance and licences** of vendored upstream data: `docs/provenance.md`. +Every scenario is graded on the world's final state and on the calls the agent made through MCP: + +| Check | Passes when | +|---|---| +| `state_ok` | The world ends in the expected state (the issue exists, the replicas changed, …) | +| `required_calls_ok` | The agent made each required API call through an MCP server | +| `forbidden_calls_ok` | The agent made none of the forbidden calls (wrong refund, deleted repo, …) | +| `answer_ok` | The written answer, when the scenario asks for one, contains what it must | + +`reward` is 1 when all four pass, otherwise 0. `outcome_code` is 0 pass, 1 fail, or 2 context +overflow: the model rejected the prompt before making any tool call, which is what happens +when the full tool list doesn't fit its context window. `api_calls` counts the agent's own calls +(the servers' startup requests are subtracted) and `mcp_bypass` counts calls that skipped MCP. + +## Determinism + +The world runs on a logical clock with seeded IDs and records every request in a journal. +`world-replay` re-issues a trial's journal against a fresh world, and the result must match byte +for byte. Tool lists are captured offline and are hash-identical across runs. + +## Scenarios + +150 scenarios in six groups (search, code, ops, cloud, business, collab): 120 use one service +and 30 cross services. Each one is generated into a Harbor task in `tasks/`. Every task scores +reward 1 with Harbor's oracle agent (which replays the solution over MCP) and 0 with the nop +agent. + +## Results so far + +- **Smoke run:** Claude Code with Claude Haiku 5.5 on OpenRouter passed 3 tasks with tool + search on and off. With every tool sent upfront, a single turn is about 820k Claude tokens + (`docs/research/smoke-2026-10-31.md`). +- **Deferred-tools arms:** router-side tool search vs Claude Code's own ToolSearch vs all tools + upfront, 324 runs over Claude Haiku 5.5 and Kimi K2.7 Code + (`docs/research/arms-2026-10-31.md`). +- **Tool payload:** tools, bytes and tokens per endpoint in `docs/research/tool-payload.md`. ## Layout | Path | What | |---|---| | `catalog/services.toml` | Source of truth for the 30 services and 32 MCP endpoints: pins, launch commands, env, ports, TLS hostnames | -| `crates/mock-core` | Shared mock library: logical clock, deterministic IDs, document store, datasets, request journal, admin API | +| `crates/mock-core` | Shared mock library: logical clock, deterministic IDs, document store, datasets, request journal, grading, admin API | | `crates/services/*` | One mock API per service | -| `crates/world` | Binary hosting every mock on its port, plus HTTPS on 443 (local CA) for fixed-hostname clients | +| `crates/world` | Binary hosting every mock on its port, plus HTTPS on 443 (local CA) for clients with fixed hostnames | | `crates/mcp-tools` | Stdio↔streamable-HTTP bridge, replica backend, MCP clients, `toolmeter` (capture, report, import) | | `crates/catalog` | Catalog parser and CLI, plus `tools-supervisor` (the tools container entrypoint) | +| `crates/scenario` | `taskgen`: turns `scenarios/` into `tasks/` | +| `crates/report` | `bench-report`: compares two runs | +| `scenarios/` | Scenario sources (`scenario.toml`, `instruction.md`, `dataset/`) | +| `tasks/` | Generated Harbor tasks; never edit by hand | | `servers/` | npm and Python lockfiles, per-server config, capture scripts, tool-list snapshots | | `images/` | `world` and `tools` Dockerfiles | -| `deploy/` | Local compose stack and the offline capture script | -| `scenarios/` | Scenario sources (`scenario.toml`, `instruction.md`, `dataset/`) | +| `deploy/` | Local compose stack, capture, oracle and smoke-run scripts | +| `docs/` | Design, authoring guide, runbook, research notes | + +## Development -## Common tasks +Needs Rust 1.99 and Docker. ```bash cargo test --workspace # all tests cargo run -p catalog --bin catalog -- validate # check the catalog +cargo run -p scenario --bin taskgen # validate scenarios, regenerate tasks/ docker build -f images/world/Dockerfile -t tool-bench-world:dev . docker build -f images/tools/Dockerfile -t tool-bench-tools:dev . @@ -44,6 +83,7 @@ docker build -f images/tools/Dockerfile -t tool-bench-tools:dev . docker compose -f deploy/compose.dev.yaml up # world + tools; MCP on localhost:7101-7132 ./deploy/capture-stack.sh # offline capture of every endpoint -> servers/snapshots + crates/world/startup-calls.json cargo run -p mcp-tools --bin toolmeter -- report # table of tools, bytes and tokens per endpoint +deploy/oracle.sh code/github/file-login-bug # oracle must score 1, nop 0 (needs Harbor) ``` The world on its own: @@ -54,6 +94,14 @@ curl localhost:8101/api/v3/repos/octo-org/app/issues curl localhost:9000/_bench/journal ``` +Images, running on Harbor and comparing runs: `docs/runbook.md`. Architecture: +`docs/superpowers/specs/2026-10-30-tool-bench-design.md`. + +## Contributing + +Adding a service or a scenario: `docs/authoring.md`. See `CONTRIBUTING.md` for the checks a pull +request must pass, and `SECURITY.md` to report a vulnerability. + ## Licence MIT (`LICENSE`). Vendored upstream data keeps its own licence: see `docs/provenance.md` and diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 00000000..2b8c53f1 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,7 @@ +# Security + +Please don't open a public issue for a vulnerability. Report it privately through GitHub's +[security advisory form](https://github.com/OpenRouterTeam/tool-bench/security/advisories/new). + +tool-bench runs third-party MCP servers against local mock APIs. The credentials in +`catalog/services.toml`, `servers/` and the scenario datasets are placeholders, not real secrets. diff --git a/docs/research/arms-2026-10-31.md b/docs/research/arms-2026-10-31.md index a1acacec..d7ef13f8 100644 --- a/docs/research/arms-2026-10-31.md +++ b/docs/research/arms-2026-10-31.md @@ -1,6 +1,6 @@ # Deferred-tools arms, 2026-10-31 -`deploy/smoke-arms.sh` runs one task under one arm; `deploy/tally-arms.py` tallies a results directory per model. 18 stratified tasks (two single-service and one cross-service task from each of the 6 groups) × 3 attempts × 3 arms, Claude Code with all 32 MCP endpoints (1,618 tools) attached, providers pinned. +`deploy/smoke-arms.sh` runs one task under one arm; `deploy/tally-arms.py` tallies a results directory per model. 18 stratified tasks (two single-service and one cross-service task from each of the 6 groups) × 3 attempts × 3 arms, Claude Code with all 32 MCP endpoints (1,595 tools) attached, providers pinned. - `universal`: `ENABLE_TOOL_SEARCH=false`; `deploy/arm-proxy.ts --universal` adds `deferred_tools {protocol: v1, profile: universal, validation: runtime, search: bm25, max_results 8}` and routes to a local router running the universal profile. - `native`: Claude Code's own client-side ToolSearch (`ENABLE_TOOL_SEARCH=true`).