Skip to content

tests(advisory): maintainer-side authoring audit — duplicate rules and routing misses - #394

Merged
MendixMau merged 3 commits into
masterfrom
tests/authoring-audit
Oct 15, 2026
Merged

tests(advisory): maintainer-side authoring audit — duplicate rules and routing misses#394
MendixMau merged 3 commits into
masterfrom
tests/authoring-audit

Conversation

@MendixMau

@MendixMau MendixMau commented Oct 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Direct lane

What changed and why (one paragraph):
A maintainer-side, advisory authoring audit, tests/advisory/authoring-audit.py (Python 3 stdlib only, never shipped: not under bin/ or project-bin/). dupes splits skills/**/*.md, agents/*.md, CLAUDE.md and README "Baseline routing" into rule-sized chunks (table rows, bullets, short paragraphs). It skips ROUTING:BEGIN/END blocks, same-file pairs and agent-to-agent pairs, ranks cross-file pairs by sparse TF-IDF cosine on 1–2-grams (inverted index, at most 3 pairs per file pair), and prints the top N that are not in tests/advisory/dupes-allow.tsv. The allowlist is seeded with the declared architect-agent ↔ workflow-structure-rules mirror and the P-labelled (pointer/tombstone/boilerplate) pairs. routing runs BM25 over the trigger text (column 3) of every bin/lib/skill-routing.tsv row against the 55 labelled phrasings in tests/advisory/routing-phrasings.tsv, prints top-1/top-3 and lists every phrasing whose expected skill is not in the top 3. It always exits 0; --strict exits 1 only when a must_hit row misses, and none is marked yet. Also in this PR: a non-blocking CI job, a golden fixture, a one-line subject-resolution fix in run-all.sh, and a pointer under CLAUDE.md authoring rule 8 ("test retrieval").

Impact

Row Answer
Process One advisory CI job (authoring-audit, continue-on-error on job and step, || true on both runs). It prints output and never gates. No human step, no stage artifact.
Dependencies Python 3 stdlib only. Shared file touched: tests/wave2/run-all.sh. Its usage-line regex now accepts .py, and tests/advisory/ sits next to #389's tests/behaviour/ in the one candidate list (origin/master merged in after #389 landed). Subject resolution is replicated for all 113 wave2 fixtures: only test-authoring-audit.sh differs from master, and none are unresolved. The tool reads only bin/lib/skill-routing.tsv and repo markdown, which always exist.
Context cost Projects: none. No routed or project-loaded file is touched. bin/render-routing.sh --check gives the same result on base (origin/master) and head: "Routing surfaces in sync with the table; all skills routed or exempted. Baseline: 73522 words (budget 80000)." Stage 5 is 68226 words / 25 files on both (lead 25820/10, ba 25842/9, architect 23929/8, mdl 30185/14, gate 22354/11, test 21296/8, review 30889/12). Toolkit-repo sessions: about +25 words in the always-loaded CLAUDE.md (the rule-8 pointer).
Behaviour NOT RUN: the only agent-read change is a two-line pointer in CLAUDE.md rule 8, a process doc that review agents read.
Runtime Measured on this runner: dupes takes 15.7 s over 7,031 chunks / 143 files (16.3 s before the review fixes); routing takes under 0.1 s. Hot paths unaffected: the CI job runs in parallel, and the fixture adds about 1 s to run-all.sh.

Field evidence. The experiment ran on master 484e7ff, with 69 hand-judged pairs (T/P/F):

  • TF-IDF cosine: 19/40 true duplicates in its top 40 and 82.5% non-false-positive. 5-shingle Jaccard was worse.

  • BM25 on trigger text: top-3 46/55. That matches or beats an LSA stand-in (the embedding model could not be downloaded).

  • Stdlib port on the same master: routing top-1 36/55 and top-3 46/55, identical to the numpy run. dupes top 40 after the allowlist: 20 judged T, 9 judged F, 11 unjudged.

  • No client data anywhere in the diff. Leak guard passes, and a whole-word grep of added lines against the maintainer's name list plus /Users/ and /home/user found nothing.

  • Size cap: 8 files, 270 lines excluding the two data TSVs and CHANGELOG.

  • Test tier: fixture tests/wave2/test-authoring-audit.sh (5 checks: exit 0, top-3 ≥ 44/55 floor on the committed phrasings, miss listing on a synthetic one-hit/one-miss file, must_hit exits 0 without --strict and 1 under --strict). Usage-line subject resolution was replicated by hand: only this fixture's resolution changes.

  • Instrument: the golden input is the real skill-routing.tsv at master and the labelled phrasings from the experiment. There are no layout or platform assumptions: the fixture uses require_py.

  • CHANGELOG.md line appended.

Checks run locally:

🤖 Generated with Claude Code

https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw

Generated by Claude Code

claude added 3 commits October 15, 2026 03:51
@claude
tests(advisory): maintainer-side authoring audit — duplicate rules an…
6e3d7ef 
…d routing misses

tests/advisory/authoring-audit.py (Python 3 stdlib only, never shipped):
- dupes: sparse TF-IDF cosine on 1-2-grams over rule-sized chunks of skills/, agents/,
  CLAUDE.md and README "Baseline routing"; allowlist seeded from the hand-judged
  pointer/tombstone/boilerplate pairs and the declared architect-agent mirror.
- routing: BM25 over skill-routing.tsv trigger text vs 55 labelled phrasings; lists top-3
  misses; --strict exits 1 only on a must_hit miss (none marked yet).

Non-blocking CI job, golden fixture tests/wave2/test-authoring-audit.sh (floor top-3 >= 44/55),
run-all.sh resolves .py subjects and tests/advisory/, CLAUDE.md rule 8 pointer.

Field evidence, master 484e7ff: routing top-1 36/55, top-3 46/55; dupes top 40 = 20 judged
true duplicates, 9 false positives, 11 unjudged; dupes 16.3 s over 7,031 chunks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
@claude
tests(advisory): review fixes for the authoring audit (#394)
4156bec 
- dupes: allowlist check runs before the per-file-pair cap, so allowlisted pairs never spend it.
- dupes: relpaths normalised to forward slashes (Windows).
- dupes-allow.tsv: four broad match substrings replaced by phrases from the allowlisted chunks;
  the same 18 pairs stay allowlisted and the top 40 is unchanged.
- fixture: miss listing proven on a synthetic one-hit/one-miss phrasings file, so it keeps
  testing after the real triggers improve; floor stays 44/55.
- checks.yml comment: ~15 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
@claude
Merge origin/master (#389) into tests/authoring-audit
5cf8c1e 
run-all.sh: tests/advisory/ joins #389's tests/behaviour/ in the one candidate list.
CHANGELOG: both Unreleased lines kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
@MendixMau
MendixMau merged commit 1512b09 into master Oct 15, 2026
2 checks passed
MendixMau pushed a commit that referenced this pull request Oct 15, 2026
@claude
Merge origin/master into feat/mxpatch-and-fix-hashes
4878984 
Master moved again during the field-run fixes (#393, #394). Clean merge;
render-routing --check re-run after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
@MendixMau MendixMau mentioned this pull request Oct 15, 2026
Merged
4 of 6 tasks
MendixMau added a commit that referenced this pull request Oct 15, 2026
@MendixMau @claude
fix(routing): rewrite nine triggers the retrieval audit showed never …
a503c30 
…fire (#391)

Trigger text (column 3) of nine skill-routing.tsv rows rewritten and re-rendered (README, ROUTING, six agent stubs). Every rendered cell is a superset of its base content words. Retrieval (BM25, 55 labelled phrasings, in-sample): full trigger top-3 46 -> 55, agent-visible stub text top-3 39 -> 47; no phrasing ranks worse.

Review: independent review MERGE (superset verified across all 44 rendered cells; numbers reproduced; render-routing --check identical totals; routing fixtures pass; merge-tree clean vs #392/#393/#394). First commit truncated stub rows before " — " (lost "Every module before it is called done" etc.); fixed in 9bcd19d. Behaviour: probe A/B on the head pack 15/15, same as baseline. Known: bug-lookup row widened to every CE error (intended); one stub-mode over-trigger (scss phrasing ranks wiring-sweep #1, learned-css #2; base had learned-css outside top 5); agent-stub routing words are not counted by render-routing --check (mdl stub +69 words).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw
@MendixMau
MendixMau deleted the tests/authoring-audit branch October 15, 2026 10:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@MendixMau @claude