new(tests/behaviour): standing behaviour suite — M/T/N arms, skill ablation, tool-call and token columns

[MendixMau]

Direct lane (changes to skills/, bug-logs/, bin/, project-bin/, project-tests/, pipelines)

What changed and why (one paragraph):
At the maintainer's request, the A/B kit used on 2026-10-15 to test the stub diet (#369/#378/#380) is now a standing suite in tests/behaviour/. build-packs.sh builds a context pack from any ref in a temporary detached worktree. Arm M is a base ref and arm T a candidate ref. With --without <skill>, T is the base pack minus one skill, which gives that skill's value as a number. Arm N is no pack: a 35-word "you are an agent writing mxcli MDL" preamble, so it measures plain reasoning. score.sh keeps the mechanical scorer and resolves everything relative to the repo (bin/mdl-scan.sh, $MXCLI or mxcli on PATH, NA without one). It adds three recorded columns: read_all, tool_calls and tokens. read_all is 1 when the agent's final reply quotes the pack's PACK-END: line, so a partial read is detected mechanically. Partial reads are counted and reported, never dropped. RUNBOOK.md gives:

• the triggers: every PR touching baseline-routed text or an agent stub; the N arm and ablations at each release cut;
• the rep counts: 5 × page, 5 × mf, and 5 probes × 3 per arm;
• the rule: T no worse than M on every scored metric, probes T ≥ M − 1;
• concurrency: up to the orchestrator's agent cap.

docs/RELEASE-PROCESS.md §4a now names the suite as the way to run the Behaviour A/B. tests/wave2/run-all.sh also resolves fixture subjects under tests/behaviour/; without that, a # usage: … tests/behaviour/score.sh line resolved to a missing score.sh and failed.

Finding from the committed run, written into RUNBOOK step 3: subagents inherit the orchestrating session's project CLAUDE.md. On 2026-10-15, 13 of 20 drafts named the host project's model file, which no pack or task contained. That contamination reached every arm equally, but it means N was not truly "no pack". The orchestrator must now start in a directory with no project CLAUDE.md.

Impact

Row	Answer
Process — human steps or gates added/removed; does anything new block; new stage artifacts	Adds one step to PRs that change baseline-routed text or an agent stub: run the suite (M = base, T = head) and paste the verdict in the Behaviour row. §4a already required an A/B there; this names the way to run it. Adds one step at each release cut: the N arm plus an ablation per baseline skill under review. Nothing blocks mechanically, and no stage artifacts change.
Dependencies — shared files touched (_common.sh, exec.sh, skill-routing.tsv, …); order vs other open PRs; new tools/platform assumptions; new artifact reads + their producer	Shared files touched: tests/wave2/run-all.sh (one extra subject-candidate directory) and docs/RELEASE-PROCESS.md §4a (one line). The suite reads bin/lib/skill-routing.sh (routing_baseline_pack), bin/runbook-slice.sh and agents/<role>-agent.md at the chosen ref, plus bin/mdl-scan.sh, all read-only. The packs it reads are produced by its own mandatory step, build-packs.sh. Tools: git worktree, cksum, POSIX awk/sed; mxcli is optional. It assumes an orchestrator that can spawn subagents with Read/Write only and report tool_uses / subagent_tokens. No ordering constraint against other open PRs.
Context cost — always-loaded words per role at the affected stage(s), base → head, and baseline total vs 80,000, pasted from bin/render-routing.sh --check on both — or "none: no routed or always-loaded file touched"	0 — none: no routed or always-loaded file touched (render-routing.sh --check exits 0, stage 5 mdl 30185/14 unchanged).
Behaviour — A/B result for agent-read text (process/preflight-skill-baseline-2026-08-31.md), or "NOT RUN: ", or "n/a: scripts only" (process docs that review agents read are agent-read text)	n/a for the suite itself: it is a test kit, nothing routes it into any agent's context. The one-line §4a pointer is process-doc text that review agents read, so per §4a that line is NOT RUN: pointer only, no rule changed.
Runtime — seconds/resources added or saved per exec, gate or check: measured, or "n/a"	Nothing on a hot path: exec, gates and checks are unchanged. A full PR run is about 50 agent reps at 2–4 agent-minutes each, roughly 100–200 agent-minutes (10–20 min wall clock at the agent cap). A release-cut N run is 10 reps of about 1 agent-minute each. build-packs.sh takes 0.4 s per pack. The new fixture takes under 2 s.

Field evidence — the 2026-10-15 runs, committed under tests/behaviour/results/ with one model fixed for every rep:

• Stub diet (chore(context): move stories and history out of always-loaded skills (T4) #369/chore(context): runbook slice for helpers; runbook drops its copy of the baseline table (stub diet T2+T3) #378/chore(context): per-role on-demand token in the routing agents column (stub diet T1) — stacked on #378 #380): 42 reps. 39 read the whole pack: 20 drafts and 19 probes. 3 were partial reads, all on the 4,822-line M pack; they are kept as read_all=0 rows. T held all eight metrics that run's rule named. quoted on the mf draft was lower (median 93 vs 95); the new every-metric rule would report that. M ran 1 probe rep each, not 3.

• First N-arm run: 5 × page and 5 × mf. The page trap was refused 0/5 without a pack, against 5/5 with either pack. The mf draft passed mxcli check 1/5, against 5/5.

• Scorer replay: the moved scorer re-scored all 49 scored outputs from both runs, using mxcli v0.26.0. Every row came out byte-identical to the original results.

• Pack replay: build-packs.sh M d16f578 reproduced that night's M pack body byte for byte.

• No client data anywhere in the diff. LEAKGUARD_ALLOW_NO_DENYLIST=1 bin/check-no-client-data.sh is clean. A whole-word grep of added lines for the client-term list and real local paths found 0 hits. The four captures were chosen from the outputs that do not name the host project. They were grepped, and the page capture was read in full.

• Size cap: not met on lines, met on files. 8 non-fixture files, 435 lines; the cap is 400. Excluded from the count: the tasks and probes in tasks/ (7 files, 140 lines of test input), the 4 captures, the 2 results files and CHANGELOG.md. Most of the overrun is score.sh at 207 lines, the 2026-10-15 scorer moved whole (185 lines before) so its replay stays byte-identical. Splitting the scorer from its runbook and fixture would ship an instrument without its proof.

• Test tier reached: T1/T2. tests/wave2/test-behaviour-score.sh tests/behaviour/score.sh gives PASS=7 both without mxcli (check=NA, not compared) and with MXCLI set (check compared). A mutation that removes one folder clause from a capture fails it. bin/check-scripts.sh, bin/check-portability.sh (covers tests/, new files clean), bin/render-routing.sh --check and bin/check-pr-discipline.sh origin/master are all clean.

• For a new/changed instrument: golden input is captured, not hand-written: 4 real agent outputs (M page, T mf, N page, N mf). Layouts: n/a, the suite reads no project tree. Platforms: Bash 3.2 constructs only, with cksum, a portable mktemp -d template and the ANSI strip done with tr -d '\033' instead of GNU sed \x1b.

• For a new skill: n/a, no skill added.

• CHANGELOG.md line appended in this PR, credited to the maintainer request.

• New bug entries: n/a.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw

---

Generated by Claude Code