feat(framework): measurement that works on five tools, each proven by a session that ran

[blafourcade]

🎯 What & why

Know what a feature cost — tokens, models, time — across Claude Code, Codex, Copilot, Cursor and OpenCode, by reading the files those tools already write. The aidd-telemetry plugin does it standalone: no CLI, no account, no endpoint, nothing leaving the machine.

🛠️ How it works

Three skills, and nothing else to install: 00-init allows measurement for a project, 01-cost answers what a period or a task consumed, 02-check says whether the chain is actually recording.

One contract, five adapters. Each tool's own file shape is mapped to one record by a per-tool adapter (claude-code-transcript.ts, codex-rollout.ts, copilot-events.ts, opencode-export.ts), against a closed allowlist in metrics-contract.md. A field not on that list cannot be stored, so no prompt, code or diff ever can be.

An unknown is never a zero. Cursor writes no token count anywhere; Copilot writes a session total and never a per-request figure. Both are reported as what they are, beside the figure, rather than as a 0 that reads as free. The same rule governs attribution: every record says how strongly it is attributed — tool-stated, journal-interval, or unattributed — and an unattributed turn stays its own row instead of being folded into the nearest step.

The sink is append-only. A stored record is never corrected in place, only reconciled on read. That is what lets a session that was read while still running be superseded later by a strictly better reading of the same turn, rather than frozen partial.

Two mirrors, pinned equal. The CLI and the plugin compute the same report from the same records, and a parity suite compares them — field values, and now also rendered row order, each guard verified against a planted divergence.

The one that mattered. Claude Code writes a line when a message starts and again when it completes, sharing one message.id. The accumulator kept the first — a placeholder carrying output_tokens: 3 where the real figure was 329. Measured across 1,604 real transcripts, that was 37% of output tokens lost, and no test saw it: the code was consistent with what it asserted about itself. The fix is validated against Claude Code's own /usage screen over 15,684 billed calls — output and cache-read agree to ~3%, and input/cache-write differ by construction, because /usage weights a 1-hour cache line by its price while a transcript counts it once.

The figure this produced. From a real three-skill chain, the breakdown against its own total:

sum of the four by_step rows   18 / 42 / 6,865 / 925,799 / 42,289
totals in the same JSON        18 / 42 / 6,865 / 925,799 / 42,289

Integer-exact on all five token fields. The fourth row is unattributed — the planning turn before the first skill call. That run is history, not a guard: no test reproduces those exact totals. What CI enforces is the invariant behind them, that every breakdown reconciles to its own total exactly, on both sides.

🧪 How to verify

cd cli && pnpm test:unit && pnpm test:integration && pnpm test:e2e
node --test "scripts/__tests__/*.test.js"
node scripts/check-cli-layering.mjs && node scripts/check-markdown-links.js

At b7141508: unit 2044, integration 608, e2e 194, plugin 518, all green; tsc, biome, layering and link checks clean. cli / Windows runs the same chain and the same three suites on windows-latest.

End to end, on a real repository:

node plugins/aidd-telemetry/skills/00-init/scripts/telemetry-switch.js on
# work as usual, then
node plugins/aidd-telemetry/skills/01-cost/scripts/telemetry-report.js report
node plugins/aidd-telemetry/skills/02-check/scripts/telemetry-check.js

⚠️ Heads-up

Found by running, not by reading. Each of these was invisible to a green suite:

• skills/_shared/ did not survive an install. Four of the five flat contracts rename every immediate child of skills/ to <plugin>-<child> (flat-paths.ts:44), so require("../../_shared/readers.js") resolved to a name that no longer existed and the reporter died at load. The install-shape test copied skills/ verbatim and could not see it. Each skill now carries its own scripts/lib/, pinned byte-identical, and a cross-skill require fails the suite.
• A directory marker does not survive a rename. The plugin declared "type": "commonjs" through four package.json files, and the hyphen-flat rename above moved skills/package.json away from the scripts it covered. Every CommonJS file it ships is now .cjs, which Node reads as CommonJS whatever the host declares, and a package.json anywhere under the plugin fails the suite. hooks/opencode-plugin.js is the one exception: OpenCode auto-discovers {plugin,plugins}/*.{ts,js} and nothing else — measured, when renaming it .mjs made OpenCode find it, log it, and never run it.
• Cursor never loads a plugin's own hooks.json. Seven declared events, three probes, headless and interactive: not one fired. The project's own .cursor/hooks.json fires normally — the obstacle was where we installed, not Cursor.
• OpenCode runs only a genuine ESM export, and auto-discovers {plugin,plugins}/*.{ts,js} — never .mjs.
• Codex will not run a hook nobody approved, and says nothing. Trust is per entry, so a renamed event inherits no approval, and a headless run never sees the prompt.
• os.homedir() never reads $HOME on Windows, it reads USERPROFILE. The plugin resolved HOME || homedir(), the CLI called bare homedir() — so a Windows run read 43,853 tokens from fixtures a POSIX run read 183,939 from.

Known limits are in the plugin's README, each with its measurement. The three that bite first: Codex needs one interactive approval; OpenCode misses a server process's first session, and opencode run is always a first session; only Claude Code's writes name a task.

Not met, under issues that are closing. #703's "a session that resolved no skill is distinguishable from one that needed none" has no implementation. #694's 2000-entry cap was measured against a synthetic tree, not the real one its bullet names. #631's evidence reads "the same skill proves both"; it is two, 01-cost and 02-check.

Unrelated reorg riding along. 12 files move from aidd_docs/plans/ to aidd_docs/tasks/ (R100 renames, no content change). Nothing in this branch depends on it; say the word and it comes out.

Left open deliberately. #702 (the install shape is reconstructed by the guard, not driven through the real translator) and #683 (hooks/lib/host.js still names tools in executable code, so a sixth tool is still two files).

🔗 Linked issue

These do not fire on this merge. GitHub auto-closes only into the default branch, and this targets next. The list is the record of what the branch delivers; the issues need closing by hand at merge time, or when next is promoted.

Closes #617, #631, #676, #680, #681, #686, #688, #693, #694, #695, #697, #698, #699, #700, #701, #703, #704, #705

✅ I certify

• I DO CERTIFY I READ EACH LINE OF THE PULL REQUEST BECAUSE I AM A SOFTWARE ENGINEER, NOT A AI PUPPY.

[blafourcade]

What to read, and why the file count lies

GitHub says 1801 files. 142 of them are this work.

The branch was cut on 16 July at chore: release main (#447). Since then main moved 115 commits and this branch moved 165, and in that window main received the CLI migration while this branch carries its own copy of it (10bdd605 fix(cli): migrate aidd-cli into framework as cli/). The three-dot diff a pull request shows is measured from the merge base, so it counts that migration twice — every file under cli/src, cli/tests, kanban and the other plugins is in the total for that reason and not because anything here touched them.

The eleven commits that are this work, from a0beac5b to 1015566f:

142 files changed, 10,518 insertions(+), 418 deletions(-)

  32  plugins/aidd-telemetry     the hooks, the three skills
  31  cli/tests                  and 29 cli/src
  28  aidd_docs/tasks            the plans, phases and measurements
  13  scripts/__tests__          the plugin's own suite
   3  docs                       what each tool can and cannot measure

Compare view for exactly those: 0509345...1015566

The base needs a decision before this leaves draft

Three ways out, none of which I will take unasked because they all rewrite or reshape someone else's branch:

1. Rebase onto current main. The duplicated migration should collapse and the diff would show its true size. It rewrites 165 commits of history.
2. Merge main in. No history rewritten, and it fixes the diff for the same reason — the merge base becomes main's tip. I dry-ran it: 90 conflicts, 63 of them add/add on the duplicated migration, 38 under cli/src and 20 under cli/tests. Resolvable, and not a thing to do quickly.
3. Leave it and review by range, using the link above.

Everything else is ready

Eleven commits, each passing the pre-commit hook without modification. The pre-push run: 252 test files, 2680 tests. Plugin scripts 365, CLI unit 1931, integration 577, e2e 178, tsc clean, biome clean, no broken markdown links.

Every per-tool claim in the body rests on a session that was actually run, and the negative results are in aidd_docs/tasks/2026_08/2026_08_22_telemetry-every-tool/measurements.md in full — including the two probes that disagreed about OpenCode and the measurement that reconciled them.

[blafourcade]

Retargeted to next, which is where it should have gone

CONTRIBUTING.md:53 says "Branch off next, target next". This was opened against main, which is the branch promote.yml rebase-merges next into — not one a feature branch belongs on. Fixed.

It also shrinks the count, because the merge base moves from 16 July to 13 August:

against main    1801 files   +164,300
against next     368 files    +45,625
this work        142 files    +10,518

Still not 142, and the reason is unchanged: the branch carries its own copy of the CLI migration that both main and next have since received, so the diff counts it twice. The comparison range in the previous comment is still the thing to read.

The base decision stands, and it is now smaller

Merging next in was dry-run against main and produced 90 conflicts. That number will differ against next and I have not re-measured it. The three options are the same — rebase, merge, or review by range — and none is mine to take on someone else's 165 commits.

Not merged, deliberately

The eight Closes fire on merge, so the backlog closes when this does. I have not merged it and will not: this is a shared repository with a release process (promote.yml rebase-merges next into main, back-merge.yml returns the other way), a draft pull request nobody has reviewed, and 165 commits of which 154 are not mine. Landing that on a shared branch is a call for whoever owns the release train, not for the author of the last eleven commits.