CodingThrust · isPANN · Jul 8, 2026 · Jul 8, 2026
diff --git a/.claude/skills/run-benchmark/SKILL.md b/.claude/skills/run-benchmark/SKILL.md
@@ -3,7 +3,7 @@ name: run-benchmark
 description: >-
   Run this repo's problem-reductions bug-finding benchmark end-to-end and produce
   out/submission.json: detect a container engine, build the runner image, configure
-  submission.env, preflight, and run the budgeted agent. Works on macOS and Linux
+  submission.env, preflight, and run the step-limited agent. Works on macOS and Linux
   (Docker or rootless Podman). Use when asked to run, test, reproduce, or smoke-test the benchmark,
   or to generate a submission for a model. NOT for `make test-unit` / pytest.
 ---
@@ -30,9 +30,7 @@ Step 6 — don't ask for intake secrets unless they're submitting.
 
 Confirm the user has:
 
-- A **model API key** and its **price per 1M tokens** (input + output) — needed for *either*
-  goal, since running the agent calls the model. Both *required*; a real run hard-fails
-  without `PRICE_IN`/`PRICE_OUT` (there is no built-in price table).
+- A **model API key** — needed for *either* goal, since running the agent calls the model.
 - **Only if submitting**: `PRB_SUBMIT_URL` + `PRB_API_KEY`, from the maintainer.
 - **git** and a **container engine** (checked next).
 
@@ -78,16 +76,13 @@ If `submission.env` is absent: `cp submission.env.example submission.env`. It's
 ```ini
 MODEL_NAME=openai/gpt-5.4    # any LiteLLM-routable name (anthropic/… openai/… openrouter/… gemini/…)
 API_KEY=sk-...               # generic; or a provider var (OPENAI_API_KEY / ANTHROPIC_API_KEY / …)
-PRICE_IN=3.0                 # USD / 1M input tokens  — REQUIRED
-PRICE_OUT=15.0               # USD / 1M output tokens — REQUIRED
 ```
 
-For a cheap smoke run (don't spend the full $20) add `MAX_RULES=1`. A **ranked** submission
-must keep `BUDGET_USD=20` and omit `MAX_RULES`.
+For a cheap smoke run add `MAX_RULES=1`. A **ranked** submission must omit `MAX_RULES`.
 
 **Agent mode** (`AGENT_MODE`, default `per-rule`): `per-rule` runs one isolated agent session
-per rule with the budget split evenly; `whole-repo` runs ONE session over the whole library
-and lets the agent enumerate and triage the rules itself under a single budget. Both produce
+per rule (35 steps each); `whole-repo` runs ONE session over the whole library (300 steps)
+and lets the agent enumerate and triage the rules itself. Both produce
 the same `out/submission.json` and are scored identically — set `AGENT_MODE=whole-repo` to try
 it. (`MAX_RULES` only applies to `per-rule`.) In `whole-repo`, the agent also writes each
 certificate to `TRAJECTORY_DIR/certs.txt` (default `/out`) as it finds it, and the trajectory
@@ -99,26 +94,25 @@ overwrite each other.
 
 **Confirm the experiment parameters with the user — don't silently default them.** These
 shape the result and the spend, so state the resolved set and get an explicit OK before
-running: **mode** (`AGENT_MODE`), **budget** (a full ranked run at `BUDGET_USD=20`, or a
-cheap smoke run via `MAX_RULES=1` / a smaller budget), and — only if they care —
-`PER_RULE_BUDGET` and `MAX_TOKENS`. Ranked runs require `BUDGET_USD=20` and no `MAX_RULES`.
+running: **mode** (`AGENT_MODE`), full ranked run vs cheap smoke run (`MAX_RULES=1`), and —
+only if they care — `MAX_TOKENS`. Ranked runs must omit `MAX_RULES`.
 
 ## Step 4 — Preflight (one tiny real API call, ~a fraction of a cent)
 
-Always run this before the full run; it validates key/endpoint/price + pred/rules through the
+Always run this before the full run; it validates key/endpoint + pred/rules through the
 exact batch code path and fails fast.
 
 - **docker**: `make preflight`
 - **podman/raw**: `<engine> run --rm --env-file submission.env problem-reductions-runner:v0.6.0 --preflight`
 
 It prints three checks (`pred binary`, `library rules`, `model call`). If any **FAIL**, read
 the detail and fix it — the `model call` line carries the real error (auth / endpoint / model
-name / pricing). Decode table in `references/env-and-troubleshoot.md`. Do not proceed on a FAIL.
+name). Decode table in `references/env-and-troubleshoot.md`. Do not proceed on a FAIL.
 
 ## Step 5 — Full run → out/submission.json
 
 **Gate**: a full run spends real money and takes a while. Restate the resolved parameters
-(model, mode, budget, any smoke caps) and get an explicit OK before you launch it.
+(model, mode, any smoke caps) and get an explicit OK before you launch it.
 
 - **docker**: `make run`
 - **podman/raw**: use the `RUN_FLAGS` from Step 1, e.g.
@@ -131,11 +125,11 @@ name / pricing). Decode table in `references/env-and-troubleshoot.md`. Do not pr
   user may need `sudo`/`chown` to move it.)
 
 The run needs network only to reach the model API. When it finishes, confirm
-`out/submission.json` exists and report the result (bugs found, spend). 
+`out/submission.json` exists and report the result (bugs found, tokens used). 
 
 ## Step 6 — Hand back, or submit
 
-`out/submission.json` now exists; report the result (bugs found, spend). Then branch on the
+`out/submission.json` now exists; report the result (bugs found, tokens used). Then branch on the
 goal from Step 0:
 
 - **Run/test locally** → done. `out/submission.json` is the deliverable; no secrets, no upload.

diff --git a/.claude/skills/run-benchmark/references/env-and-troubleshoot.md b/.claude/skills/run-benchmark/references/env-and-troubleshoot.md
@@ -13,20 +13,15 @@ these as env vars (CLI flags would override, but the skill uses the env-file).
 |---|---|---|
 | `MODEL_NAME` | LiteLLM-routable model name (`anthropic/…`, `openai/…`, `openrouter/…`, `gemini/…`, or `openai/<m>` + `API_BASE`) | run hard-errors: "`--model (or env MODEL_NAME) is required`" |
 | API key | `API_KEY` (generic) **or** a provider var (`OPENAI_API_KEY`/`ANTHROPIC_API_KEY`/`OPENROUTER_API_KEY`/`GEMINI_API_KEY`) — provider vars pass straight through to LiteLLM | not checked in Python; surfaces at the `model call` preflight as an auth error |
-| `PRICE_IN`, `PRICE_OUT` | USD / 1M input & output tokens; spend = tokens × price (the $20 cap basis) | **must be given together**; a real run hard-errors if absent — there is deliberately no built-in price table |
 
 ### Optional (defaults shown; uncomment only to change)
 | Var | Default | Use |
 |---|---|---|
-| `PRICE_CACHE_READ` / `PRICE_CACHE_WRITE` | 0 | prompt-caching models |
 | `API_BASE` | — | OpenAI-compatible endpoint (OpenRouter/gateway/vLLM/Azure) |
 | `MODEL_KWARGS` | — | JSON object of extra litellm kwargs (`custom_llm_provider`, `api_version`, `extra_headers`…). Invalid JSON / non-object errors at startup |
-| `BUDGET_USD` | 20 | must be **20 to be ranked** (not enforced by the runner; unrankable otherwise) |
-| `PER_RULE_BUDGET` | 0.5 | per-rule cost cap |
-| `SAFETY_MARGIN` | 1.0 | USD held back so the budget-crossing call stays under cap |
 | `MAX_TOKENS` | 8192 | per-call output ceiling |
 | `MAX_RULES` | all | cap rules attempted — **smoke runs only**; omit for a ranked run (per-rule only) |
-| `AGENT_MODE` | `per-rule` | `per-rule` (isolated session/rule, budget split evenly) or `whole-repo` (ONE session, the agent triages the rules itself) |
+| `AGENT_MODE` | `per-rule` | `per-rule` (isolated session per rule, 35 steps each) or `whole-repo` (ONE session, 300 steps, the agent triages the rules itself) |
 | `TRAJECTORY_DIR` | `OUTPUT`'s dir (`/out`) | where **whole-repo** persists the trajectory + the durable incremental cert log (`certs.txt`); the agent writes each certificate here the moment it finds it, so an early-stop/crash still leaves the found bugs on disk |
 | `AGENT_CONFIG` / `AGENT_STRATEGY_FILE` | bundled | bring-your-own prompt; the files must be **mounted** into the container (`-v "$PWD/cfg:/cfg"`) and the path given as a container path |
 | `SUBMITTED_BY` | — | your handle, recorded in the envelope |
@@ -40,8 +35,6 @@ MODEL_NAME=openai/my-model
 API_BASE=https://my-gateway.example/v1
 API_KEY=...
 MODEL_KWARGS={"custom_llm_provider":"openai"}
-PRICE_IN=1.5
-PRICE_OUT=6.0
 ```
 
 ## Preflight failure decoding
@@ -53,7 +46,7 @@ checks and exits non-zero if any fail. It never raises — it prints `PASS`/`FAI
 |---|---|---|
 | **pred binary** | pred missing or version ≠ pinned | should always pass inside the image; a FAIL = broken/overridden image or a wrong `EXPECTED_PRED_VERSION`. Rebuild at the right `PR_REF` |
 | **library rules** | no `.rs` rules under `REPO_DIR/src/rules` | source tree not copied / `REPO_DIR` overridden. Rebuild the image; don't set `REPO_DIR` |
-| **model call** | the real error (spends ~$0.0001) — its detail names the exception type | **auth error** → bad/missing key; **connection error** → wrong `API_BASE`/endpoint; **routing/model-not-found** → wrong `MODEL_NAME`; **pricing** → `PRICE_IN`/`PRICE_OUT`. Fix that line in `submission.env` and rerun preflight |
+| **model call** | the real error (spends ~$0.0001) — its detail names the exception type | **auth error** → bad/missing key; **connection error** → wrong `API_BASE`/endpoint; **routing/model-not-found** → wrong `MODEL_NAME`. Fix that line in `submission.env` and rerun preflight |
 
 Only proceed to the full run when preflight prints `Preflight PASSED`.
 

diff --git a/.github/scripts/check_aggregate.py b/.github/scripts/check_aggregate.py
@@ -15,9 +15,9 @@
 from pathlib import Path
 
 ALLOWED_KEYS = {
-    "model", "library_commit", "budget_cap", "bugs_found", "rules_tested",
-    "total_cost_usd", "total_tokens_k", "efficiency_bugs_per_ktok",
-    "efficiency_bugs_per_dollar", "submitted_by", "placeholder",
+    "model", "library_commit", "bugs_found", "rules_tested",
+    "total_tokens_k", "usage_totals", "efficiency_bugs_per_ktok",
+    "submitted_by", "placeholder",
     # per-submission entry files (site/results/<slug>.json) also carry provenance tags
     "timestamp", "submission_id",
 }

diff --git a/.github/workflows/score-from-r2.yml b/.github/workflows/score-from-r2.yml
@@ -135,7 +135,7 @@ jobs:
               echo "updated PR #$EXISTING ($slug)"
             else
               gh pr create --base "${{ github.ref_name }}" --head "$BR" --title "$TITLE" \
-                --body "One submission's public entry (\`$dest\`) — aggregate only (counts / cost / tokens / efficiency), no certificates or rule identities (guarded by \`check_aggregate.py\`). The full certificate stays private in R2. Merging rebuilds \`site/results.json\` from all entries and deploys via publish-on-merge. This PR is scoped to a single submission; merge/close/revert it independently."
+                --body "One submission's public entry (\`$dest\`) — aggregate only (counts / tokens / efficiency), no certificates or rule identities (guarded by \`check_aggregate.py\`). The full certificate stays private in R2. Merging rebuilds \`site/results.json\` from all entries and deploys via publish-on-merge. This PR is scoped to a single submission; merge/close/revert it independently."
             fi
             opened=$((opened+1))
           done

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -1,7 +1,7 @@
 # Submitting a model run
 
-The benchmark gives every model the **same $20 API budget** and asks: how many distinct
-reduction-rule bugs can it find?
+The benchmark gives every model the **same step-limited agent session** and asks: how many
+distinct reduction-rule bugs can it find?
 
 ```
   make run ─▶ submission.json ─▶ python -m benchmark.submit  ──▶  private store (R2)
@@ -18,8 +18,8 @@ recomputed by `pred`.
 ## 1. Produce a `submission.json` (dockerized runner)
 
 The runner image bundles the `pred` binary, the agent stack (mini-swe-agent + LiteLLM),
-and the problem-reductions source pinned at `v0.6.0`. LiteLLM enforces the budget across
-whatever provider key you supply.
+and the problem-reductions source pinned at `v0.6.0`. Any LiteLLM-routable provider key
+works.
 
 The **target library version is not hardcoded** — it tracks the benchmark. The single knob
 is the `PR_REF` build arg (a tag or commit of problem-reductions); the image bakes the
@@ -39,23 +39,21 @@ All run config goes in **one env-file** so you don't juggle a dozen `-e` flags.
 template, fill the two required lines, and run:
 
 ```bash
-cp submission.env.example submission.env   # then edit the REQUIRED lines (model, key, price)
+cp submission.env.example submission.env   # then edit the REQUIRED lines (model, key)
 mkdir -p out
 docker run --rm --env-file submission.env -v "$PWD/out:/out" \
   problem-reductions-runner:v0.6.0
 # → ./out/submission.json      (or just: make run)
 ```
 
-The **required lines are model + API key + price** (`MODEL_NAME`, a key, `PRICE_IN`/`PRICE_OUT`).
-Everything else in the template has a sane default — uncomment only what you need. The knobs,
-by tier:
+The **required lines are model + API key** (`MODEL_NAME`, a key). Everything else in the
+template has a sane default — uncomment only what you need. The knobs, by tier:
 
 | Tier | Vars | When |
 |---|---|---|
-| **Required** | `MODEL_NAME`, one API key (`ANTHROPIC_API_KEY` / `OPENAI_API_KEY` / generic `API_KEY`), `PRICE_IN`, `PRICE_OUT` | always — see the price note below |
-| **Pricing (caching)** | `PRICE_CACHE_READ`, `PRICE_CACHE_WRITE` | prompt-caching models |
+| **Required** | `MODEL_NAME`, one API key (`ANTHROPIC_API_KEY` / `OPENAI_API_KEY` / generic `API_KEY`) | always |
 | **Non-standard provider** | `API_BASE`, `API_KEY`, `MODEL_KWARGS` (JSON of extra litellm kwargs: `api_version`, `custom_llm_provider`, `extra_headers`, …) | OpenRouter / gateway / local vLLM / Azure |
-| **Budget (defaults = ranked config)** | `BUDGET_USD=20`, `PER_RULE_BUDGET=0.5`, `SAFETY_MARGIN=1`, `MAX_TOKENS=8192`, `MAX_RULES` | quick test runs / tuning only |
+| **Limits (defaults = ranked config)** | `MAX_TOKENS=8192`, `MAX_RULES` | quick test runs / tuning only |
 | **Custom prompt** | `AGENT_CONFIG`, `AGENT_STRATEGY_FILE` (mount the files too) | bring your own bug-hunting prompt |
 | **Version pins** | `EXPECTED_PRED_VERSION` (empty disables), `EXPECTED_PRED_COMMIT` | debugging only — baked from the image build |
 
@@ -64,14 +62,10 @@ by tier:
 provider. (`REPO_DIR` / `OUTPUT` are container-internal and already defaulted; you don't set
 them.)
 
-> Why you pass the price (and why it's required): you pay your own bill at your own rate, so
-> you set it. The runner computes spend from raw token counts × your price rather than
-> trusting the gateway's dollar figure (which can be stale or wrong — LiteLLM $0-pricing
-> incidents, Anthropic prompt-cache mis-pricing ~10×), so `$20` is a real cap. There is
-> deliberately **no built-in price table**: a wrong default would silently mis-meter the
-> budget, so a real run fails fast without `PRICE_IN`/`PRICE_OUT`. The backend re-verifies
-> bugs regardless and ranks on **bugs/Ktok** (token counts are auditable); self-reported
-> dollars are advisory only.
+> Runs are bounded by the agent step limit (35 steps per rule; 300 steps for a whole-repo
+> session), not by a dollar budget — you pay your own bill. Raw token counts are recorded
+> and travel in the submission (`usage_totals`); ranking is by **confirmed bugs**, with
+> **bugs/Ktok** as the efficiency tie-break.
 
 For example, a non-standard endpoint in `submission.env`:
 
@@ -80,8 +74,6 @@ MODEL_NAME=openai/my-model
 API_BASE=https://my-gateway.example/v1
 API_KEY=...
 MODEL_KWARGS={"custom_llm_provider":"openai"}
-PRICE_IN=1.5
-PRICE_OUT=6.0
 ```
 
 Equivalently with raw `-e` flags (the env-file just bundles these):
@@ -92,7 +84,6 @@ docker run --rm \
   -e API_BASE=https://my-gateway.example/v1 \
   -e API_KEY=$MY_KEY \
   -e MODEL_KWARGS='{"custom_llm_provider":"openai","extra_headers":{"X-Org":"acme"}}' \
-  -e PRICE_IN=1.5 -e PRICE_OUT=6.0 -e BUDGET_USD=20 \
   -v "$PWD/out:/out" \
   problem-reductions-runner:v0.6.0
 ```
@@ -108,8 +99,8 @@ docker run --rm --env-file submission.env \
 # For a full prompt rewrite instead, mount a config.yaml and set AGENT_CONFIG=/cfg/config.yaml.
 ```
 
-**Before the full run, validate your config** with one tiny real call (a fraction of a cent)
-so a bad key / wrong endpoint / missing price surfaces now, not 20 rules in:
+**Before the full run, validate your config** with one tiny real call so a bad key / wrong
+endpoint surfaces now, not 20 rules in:
 
 ```bash
 make preflight        # docker run --env-file submission.env <image> --preflight
@@ -207,14 +198,13 @@ would hide. The round-trip judging itself is explained in the [README](README.md
 - Counterexamples are **deterministically re-checkable** — we don't even need a hidden
   answer key; a bug either violates the rule under `pred` or it doesn't.
 - Distinct-rule de-duplication caps the count at one per rule.
-- The $20 budget is enforced inside the runner by recomputing spend from raw token usage ×
-  your declared price (not the gateway's self-reported dollars), held back by a safety
-  margin and bounded per call by `MAX_TOKENS`; the backend cross-checks reported spend.
+- Sessions are bounded by the agent step limit and per call by `MAX_TOKENS`; token totals
+  travel as raw 4-bucket counts (`usage_totals`) the backend recomputes `total_tokens_k` from.
 
 ## Status: validated against a live model
 
 The runner pipeline is unit-tested end-to-end with `FakeRunner` + the certificate fixtures
 **and** has been exercised against a live model API (a DeepSeek OpenAI-compatible endpoint
-via `MODEL_NAME=openai/<model>` + `API_BASE`): preflight passes, and a real budgeted run
-drives the agent across a rule and emits a schema-valid `submission.json`. PR scoring and
-GitHub Pages publishing are live; full `$20` runs are the remaining step.
+via `MODEL_NAME=openai/<model>` + `API_BASE`): preflight passes, and a real run drives the
+agent across a rule and emits a schema-valid `submission.json`. PR scoring and GitHub Pages
+publishing are live; full official runs are the remaining step.
diff --git a/Makefile b/Makefile
@@ -8,7 +8,7 @@
 #   preflight            Validate submission.env with one tiny real call before a full run
 #   run                  Run the benchmark via Docker → out/submission.json (does NOT upload)
 #
-# Model + key + price for the real run live in submission.env (any provider — see
+# Model + key for the real run live in submission.env (any provider — see
 # submission.env.example); preflight/submission read it via --env-file. REPO_DIR is only
 # for the local-clone targets (audit).
 
@@ -47,14 +47,14 @@ runner-build:
 	  --build-arg PR_REF=$(PR_REF) -t $(IMAGE) .
 
 ## Preflight: validate submission.env with one tiny real API call + pred/rules checks,
-## BEFORE committing to a full $20 run. Spends a fraction of a cent. (The no-API wiring of
+## BEFORE committing to a full run. Makes one tiny real API call. (The no-API wiring of
 ## the runner itself is covered by the pytest suite, not a make target.)
 preflight:
 	@if [ ! -f "$(ENV_FILE)" ]; then \
 	  echo "No $(ENV_FILE) — copy submission.env.example and fill it in first"; exit 1; fi
 	docker run --rm --env-file "$(ENV_FILE)" $(IMAGE) --preflight
 
-## Run the budgeted bug-finding agent via Docker → writes ./out/submission.json.
+## Run the bug-finding agent via Docker → writes ./out/submission.json.
 ## This RUNS the benchmark locally; it does NOT submit — submitting is a separate step
 ## (open a GitHub PR adding the file, see CONTRIBUTING.md). Config lives in submission.env
 ## (copy submission.env.example); run `make preflight` first to validate it.
@@ -118,4 +118,4 @@ help:
 	@echo ""
 	@echo "Variables:"
 	@echo "  REPO_DIR=$(REPO_DIR)"
-	@echo "  ENV_FILE=$(ENV_FILE)  (model/key/price for preflight + submission)"
+	@echo "  ENV_FILE=$(ENV_FILE)  (model/key for preflight + submission)"