docs: public README, CONTRIBUTING and SECURITY

CONTRIBUTING.md

@@ -0,0 +1,27 @@
+# Contributing
+
+Start with `docs/authoring.md`. It covers what a mock service must implement, how to write a
+scenario (instruction, dataset, expectations, oracle solution) and the realism and determinism
+rules both must follow.
+
+## Before opening a pull request
+
+```bash
+cargo fmt --all -- --check
+cargo clippy --workspace --all-targets -- -D warnings
+cargo test --workspace
+cargo run -q -p scenario --bin taskgen          # validates every scenario, regenerates tasks/
+```
+
+- Commit the regenerated `tasks/` together with the change that caused it. CI runs
+  `taskgen --check` and fails when `tasks/` is stale. Image tags are content hashes, so any
+  change under `crates/`, `scenarios/`, `servers/`, the catalog or the image Dockerfiles changes
+  them.
+- For a new or changed scenario, run `deploy/oracle.sh <scenario-id>` (needs Harbor and Docker):
+  the oracle agent must score 1 and the nop agent 0.
+- Don't edit files under `tasks/` by hand.
+
+## Data
+
+Datasets are invented. Never add real customer, account or personal data, and never add a real
+credential: tokens in the catalog and datasets are placeholders the mocks accept.

README.md

@@ -1,49 +1,89 @@
 # tool-bench
 
-A deterministic benchmark for tool calling with very large tool sets: 30 real services, about
-1,500 MCP tools, and more than 600k tokens of tool definitions. Agents run through Harbor on the
-OpenRouter benchmark flow and call real (pinned) MCP servers. Those servers talk to mock APIs
-written in Rust, which answer like the real services from curated per-scenario datasets.
+A deterministic benchmark for tool calling with very large tool sets. Each trial attaches 30
+services through 32 MCP endpoints, 1,595 tools and about 727k tokens of tool definitions, then
+asks the agent to do one realistic task: file an issue, scale a deployment, refund an order
+across Zendesk and Stripe.
 
-Design: `docs/superpowers/specs/2026-10-30-tool-bench-design.md`. Operating it: `docs/runbook.md`.
+The MCP servers are the real, pinned upstream servers (GitHub, Kubernetes, AWS, Stripe, …).
+They talk to mock APIs written in Rust that answer like the real services from a curated
+dataset per scenario. Nothing leaves the trial except model traffic, and every run of a scenario
+starts from the same world.
 
-## Status
+## What it measures
 
-- **30 services, 32 MCP endpoints, 1,595 tools, ~727k o200k tokens** of tool definitions (`docs/research/tool-payload.md`), captured offline and hash-identical across runs.
-- **150 scenarios** (120 single-service, 30 cross-service), each a generated Harbor task in `tasks/`. Every one scores reward 1 with Harbor's oracle agent (replaying the solution over MCP) and 0 with the nop agent.
-- **Determinism:** logical clock, seeded IDs, recorded journal; `world-replay` re-issues a trial's journal against a fresh world and must match byte for byte.
-- **Model smoke run:** 3 tasks × 2 modes with Claude Code + Claude Haiku 5.5 via Ori/OpenRouter, all reward 1.0 (`docs/research/smoke-2026-10-31.md`, `deploy/smoke-ori.sh`). Full payload ≈ 820k Claude tokens per turn.
-- **Deferred-tools arms:** universal vs native ToolSearch vs all-tools-upfront, 324 runs over Claude Haiku 5.5 and Kimi K2.7 Code (`docs/research/arms-2026-10-31.md`, `deploy/smoke-arms.sh`).
-- **Argo:** images are published on merge to `main` (`docs/runbook.md`); the Harbor runner registers this directory's `tasks/`.
-- **Provenance and licences** of vendored upstream data: `docs/provenance.md`.
+Every scenario is graded on the world's final state and on the calls the agent made through MCP:
+
+| Check | Passes when |
+|---|---|
+| `state_ok` | The world ends in the expected state (the issue exists, the replicas changed, …) |
+| `required_calls_ok` | The agent made each required API call through an MCP server |
+| `forbidden_calls_ok` | The agent made none of the forbidden calls (wrong refund, deleted repo, …) |
+| `answer_ok` | The written answer, when the scenario asks for one, contains what it must |
+
+`reward` is 1 when all four pass, otherwise 0. `outcome_code` is 0 pass, 1 fail, or 2 context
+overflow: the model rejected the prompt before making any tool call, which is what happens
+when the full tool list doesn't fit its context window. `api_calls` counts the agent's own calls
+(the servers' startup requests are subtracted) and `mcp_bypass` counts calls that skipped MCP.
+
+## Determinism
+
+The world runs on a logical clock with seeded IDs and records every request in a journal.
+`world-replay` re-issues a trial's journal against a fresh world, and the result must match byte
+for byte. Tool lists are captured offline and are hash-identical across runs.
+
+## Scenarios
+
+150 scenarios in six groups (search, code, ops, cloud, business, collab): 120 use one service
+and 30 cross services. Each one is generated into a Harbor task in `tasks/`. Every task scores
+reward 1 with Harbor's oracle agent (which replays the solution over MCP) and 0 with the nop
+agent.
+
+## Results so far
+
+- **Smoke run:** Claude Code with Claude Haiku 5.5 on OpenRouter passed 3 tasks with tool
+  search on and off. With every tool sent upfront, a single turn is about 820k Claude tokens
+  (`docs/research/smoke-2026-10-31.md`).
+- **Deferred-tools arms:** router-side tool search vs Claude Code's own ToolSearch vs all tools
+  upfront, 324 runs over Claude Haiku 5.5 and Kimi K2.7 Code
+  (`docs/research/arms-2026-10-31.md`).
+- **Tool payload:** tools, bytes and tokens per endpoint in `docs/research/tool-payload.md`.
 
 ## Layout
 
 | Path | What |
 |---|---|
 | `catalog/services.toml` | Source of truth for the 30 services and 32 MCP endpoints: pins, launch commands, env, ports, TLS hostnames |
-| `crates/mock-core` | Shared mock library: logical clock, deterministic IDs, document store, datasets, request journal, admin API |
+| `crates/mock-core` | Shared mock library: logical clock, deterministic IDs, document store, datasets, request journal, grading, admin API |
 | `crates/services/*` | One mock API per service |
-| `crates/world` | Binary hosting every mock on its port, plus HTTPS on 443 (local CA) for fixed-hostname clients |
+| `crates/world` | Binary hosting every mock on its port, plus HTTPS on 443 (local CA) for clients with fixed hostnames |
 | `crates/mcp-tools` | Stdio↔streamable-HTTP bridge, replica backend, MCP clients, `toolmeter` (capture, report, import) |
 | `crates/catalog` | Catalog parser and CLI, plus `tools-supervisor` (the tools container entrypoint) |
+| `crates/scenario` | `taskgen`: turns `scenarios/` into `tasks/` |
+| `crates/report` | `bench-report`: compares two runs |
+| `scenarios/` | Scenario sources (`scenario.toml`, `instruction.md`, `dataset/`) |
+| `tasks/` | Generated Harbor tasks; never edit by hand |
 | `servers/` | npm and Python lockfiles, per-server config, capture scripts, tool-list snapshots |
 | `images/` | `world` and `tools` Dockerfiles |
-| `deploy/` | Local compose stack and the offline capture script |
-| `scenarios/` | Scenario sources (`scenario.toml`, `instruction.md`, `dataset/`) |
+| `deploy/` | Local compose stack, capture, oracle and smoke-run scripts |
+| `docs/` | Design, authoring guide, runbook, research notes |
+
+## Development
 
-## Common tasks
+Needs Rust 1.99 and Docker.
 
 ```bash
 cargo test --workspace                                    # all tests
 cargo run -p catalog --bin catalog -- validate            # check the catalog
+cargo run -p scenario --bin taskgen                       # validate scenarios, regenerate tasks/
 
 docker build -f images/world/Dockerfile -t tool-bench-world:dev .
 docker build -f images/tools/Dockerfile -t tool-bench-tools:dev .
 
 docker compose -f deploy/compose.dev.yaml up              # world + tools; MCP on localhost:7101-7132
 ./deploy/capture-stack.sh                                 # offline capture of every endpoint -> servers/snapshots + crates/world/startup-calls.json
 cargo run -p mcp-tools --bin toolmeter -- report          # table of tools, bytes and tokens per endpoint
+deploy/oracle.sh code/github/file-login-bug               # oracle must score 1, nop 0 (needs Harbor)
 ```
 
 The world on its own:
@@ -54,6 +94,14 @@ curl localhost:8101/api/v3/repos/octo-org/app/issues
 curl localhost:9000/_bench/journal
 ```
 
+Images, running on Harbor and comparing runs: `docs/runbook.md`. Architecture:
+`docs/superpowers/specs/2026-10-30-tool-bench-design.md`.
+
+## Contributing
+
+Adding a service or a scenario: `docs/authoring.md`. See `CONTRIBUTING.md` for the checks a pull
+request must pass, and `SECURITY.md` to report a vulnerability.
+
 ## Licence
 
 MIT (`LICENSE`). Vendored upstream data keeps its own licence: see `docs/provenance.md` and

SECURITY.md

@@ -0,0 +1,7 @@
+# Security
+
+Please don't open a public issue for a vulnerability. Report it privately through GitHub's
+[security advisory form](https://github.com/OpenRouterTeam/tool-bench/security/advisories/new).
+
+tool-bench runs third-party MCP servers against local mock APIs. The credentials in
+`catalog/services.toml`, `servers/` and the scenario datasets are placeholders, not real secrets.

docs/research/arms-2026-10-31.md

@@ -1,6 +1,6 @@
 # Deferred-tools arms, 2026-10-31
 
-`deploy/smoke-arms.sh` runs one task under one arm; `deploy/tally-arms.py` tallies a results directory per model. 18 stratified tasks (two single-service and one cross-service task from each of the 6 groups) × 3 attempts × 3 arms, Claude Code with all 32 MCP endpoints (1,618 tools) attached, providers pinned.
+`deploy/smoke-arms.sh` runs one task under one arm; `deploy/tally-arms.py` tallies a results directory per model. 18 stratified tasks (two single-service and one cross-service task from each of the 6 groups) × 3 attempts × 3 arms, Claude Code with all 32 MCP endpoints (1,595 tools) attached, providers pinned.
 
 - `universal`: `ENABLE_TOOL_SEARCH=false`; `deploy/arm-proxy.ts --universal` adds `deferred_tools {protocol: v1, profile: universal, validation: runtime, search: bm25, max_results 8}` and routes to a local router running the universal profile.
 - `native`: Claude Code's own client-side ToolSearch (`ENABLE_TOOL_SEARCH=true`).