Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ follow the cross-pointers in their index or take the next matching seeded domain
| [databases](wiki/databases/index.md) | **seeded** | Designing schemas/tables/keys, choosing or evaluating indexes, writing or optimizing queries, choosing transaction/isolation behavior, surveying live data to derive a rule, verifying additive migrations |
| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, call-site enumeration before a contract change, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, consuming LLM APIs (completion validation, context budgeting), authoring agent-facing artifacts (binding instruction text, agent tool-surface granularity/parity), MAPE-aligned point-prediction calibration, consuming external-API responses, externally-owned defaults, object-storage references, sync-vs-async integration choice, WebSocket/SSE connection lifecycle) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps, packaging data files with `importlib.resources`) |
| [frontend](wiki/frontend/index.md) | **seeded** | Web UI code: state placement, rendering performance, in-UI data fetching (races, infinite scroll), auth token handling, forms, XSS-safe output, accessibility, agent-facing tool surfaces (WebMCP) |
| [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, rollout/rollback strategy, observability (logs/metrics/alerting), per-environment/path-valued config, multi-agent orchestration (worker liveness signals, shared run state, tmux pane delivery, completion gates, worktree-isolated workers, autonomous ask-vs-rule decisions) |
| [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, rollout/rollback strategy, observability (logs/metrics/alerting), per-environment/path-valued config, multi-agent orchestration (worker liveness signals, shared run state, tmux pane delivery, completion gates, worktree-isolated workers, autonomous ask-vs-rule decisions, session context/token budgeting) |
| [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, test-before-code ordering, cases/assertions, cross-layer effect scoping, test data, mock decisions, flaky tests (release-process quality → qa) |
| [qa](wiki/qa/index.md) | **seeded** | Release-quality process: release gates, regression scoping, bug reports, severity/priority triage, evidence for completion claims, acting on code-review feedback, adversarial review of high-risk diffs, exploratory testing (guarded-path coverage, override matrices), scope-purity gates, sourcing deliverable documents from generated artifacts, verifying the quantitative claims in a document before publishing it, automated verification of document deliverables (spec/RFC gates) (writing automated test code → testing) |
| [debugging](wiki/debugging/index.md) | **seeded** | Diagnosing a failure — finding what is wrong and why: reproducing, bisection, hypothesis testing, traces/logs, intermittent failures (fixing the diagnosed fault → its owning domain) |
Expand Down
23 changes: 23 additions & 0 deletions skills/orchestrate/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,20 @@ even if they can't re-read the config. A role is a tool injected into one step,
never a loop: do not map a role to an implement/verify-loop tool or another
orchestrator (that nests loops); there is no `implement` role.

## Coordinator token budget
Accumulated context, not turn count, is what drives a run's cost — every call
resends the full history at the cached rate, so cost per turn grows as the run
goes on. Three things the coordinator itself controls: (a) never `Read` a
screenshot or other image directly into this session — delegate the visual
check to a subagent that returns a text verdict instead; (b) bound every pane
capture and status read (`tail -N` on tmux panes, one-line `jq` filters on
status JSON) instead of pulling full output into context; (c) at Gate 1 and
Gate 2 — the two points where a human is already present — if the run has gone
long, tell the user this is a safe `/compact` point (the coordinator cannot
compact its own session). Basis:
`wiki/infrastructure/agent-orchestration/session-context-token-budget.md`
(directives 2–4).

## Preflight
Run `${CLAUDE_PLUGIN_ROOT}/hooks/preflight.sh` to resolve git/tmux/jq paths and
surface any missing CLI. **git, tmux, and jq are all required** — if any is
Expand Down Expand Up @@ -699,6 +713,15 @@ final step is the same: `LO_RUN_ID=<run-id> scripts/safe-cleanup.sh teardown
artifacts to `archive-<date>-<runid>/`. A run that ends without teardown is
exactly the leak `list-orphans --stale` exists to find.

After teardown, audit the run's token efficiency: `sh {SKILL}/scripts/token-report.sh
--cwd <root> .` for the coordinator's own session, plus one more `--cwd
<worktree>` per run worktree (each worker's transcripts live under its own
cwd). Put the table and any `warn:` lines in the final run report — a
peak-context warning is the signal to split tasks or delegate more next run.
Exit 0 is the report, exit 2 a usage error, exit 4 a malformed `LO_CTX_WARN`.
Basis: `wiki/infrastructure/agent-orchestration/session-context-token-budget.md`
(directive 1).

## Re-entry (resume)
On re-invocation with no context, measure real state first: `git worktree list`,
each `.orchestration/status/*.json` phase, and which `briefs/plans/reviews/`
Expand Down
202 changes: 202 additions & 0 deletions skills/orchestrate/scripts/token-report.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,202 @@
#!/bin/sh
# token-report.sh — per-session + total token usage report from Claude Code
# session transcript JSONL files (Claude Code writes these under
# ~/.claude/projects/<escaped-cwd>/). Read at orchestration run teardown to
# audit a run's token efficiency: calls, output tokens, cache hit %, and
# final/peak context per session.
#
# usage: token-report.sh [--cwd <project-dir>] <file-or-dir>...
# <file-or-dir> a .jsonl transcript, or a directory (its *.jsonl
# files, non-recursive). At least one is required.
# An ABSOLUTE file-or-dir is used as-is; a RELATIVE
# one is resolved against the --cwd mapping below (or
# taken as relative to the caller's cwd if --cwd was
# not given).
# --cwd <project-dir> an absolute project/worktree path, mapped to
# ~/.claude/projects/<escaped> — the directory Claude
# Code writes that project's session transcripts
# under. <escaped> is <project-dir> with every `/`
# and `.` replaced by `-`. Typical use at teardown:
# `token-report.sh --cwd "$PWD" .`
#
# exit 0 report printed: table on stdout, peak-context warnings on stderr
# exit 2 usage error — no sources given, a source path that does not
# exist, or (when --cwd is given) its mapped directory does not
# exist (stderr names the attempted path)
# exit 4 malformed LO_CTX_WARN (non-numeric or <= 0) — refused, never
# silently defaulted
# exit 127 jq not found
#
# metrics (one row per session = one .jsonl source file):
# calls count of JSONL lines carrying a `.message.usage` object. A
# line that fails to parse as JSON, or parses but has no
# `.message.usage`, is skipped — not an error.
# out sum of output_tokens across those lines
# hit% sum(cache_read_input_tokens) /
# sum(input_tokens + cache_read_input_tokens +
# cache_creation_input_tokens) * 100, one decimal;
# "0.0" when the denominator is 0
# final_ctx input+cache_read+cache_creation of the LAST usage line
# peak_ctx max input+cache_read+cache_creation across usage lines
# TOTAL row: calls/out are sums across sessions; hit% is recomputed from
# the summed token counts (not averaged); final_ctx/peak_ctx are the MAX
# across sessions (not summed).
#
# env:
# LO_CTX_WARN peak-context warning threshold (default 300000). After the
# table, every session whose peak_ctx >= this threshold gets
# one `warn: ...` line on stderr. Empty/unset = default; a
# non-numeric or <= 0 value is refused at exit 4 rather than
# silently falling back to the default.
set -eu

JQ=$(command -v jq) || { echo "token-report: jq not found" >&2; exit 127; }

usage() { echo "usage: token-report.sh [--cwd <project-dir>] <file-or-dir>..." >&2; }

# ---- --cwd: map an absolute project path to its transcript directory ------
cwd_map=""
if [ "${1:-}" = "--cwd" ]; then
if [ $# -lt 2 ]; then usage; exit 2; fi
proj="$2"; shift 2
escaped=$(printf '%s' "$proj" | tr '/.' '-')
cwd_map="$HOME/.claude/projects/$escaped"
if [ ! -d "$cwd_map" ]; then
echo "token-report: --cwd mapped directory not found: '$cwd_map' (from '$proj')" >&2
exit 2
fi
fi

if [ $# -lt 1 ]; then usage; exit 2; fi

# ---- LO_CTX_WARN: default 300000; refuse (exit 4) rather than silently
# default on a non-numeric or non-positive value ----------------------------
warn_threshold="${LO_CTX_WARN:-}"
if [ -z "$warn_threshold" ]; then
warn_threshold=300000
else
case "$warn_threshold" in
*[!0-9]*)
echo "token-report: invalid LO_CTX_WARN '$warn_threshold' (must be a positive integer)" >&2
exit 4 ;;
esac
if [ "$warn_threshold" -le 0 ]; then
echo "token-report: invalid LO_CTX_WARN '$warn_threshold' (must be > 0)" >&2
exit 4
fi
fi

# ---- resolve each positional arg to one or more .jsonl source files -------
# $1 = raw arg -> stdout: one resolved path per line. Exits 2 if it does not
# exist. A directory yields its immediate *.jsonl files (sorted, non-recursive).
resolve_source() {
arg="$1"
case "$arg" in
/*) path="$arg" ;;
*) if [ -n "$cwd_map" ]; then path="$cwd_map/$arg"; else path="$arg"; fi ;;
esac
if [ ! -e "$path" ]; then
echo "token-report: source not found: '$path'" >&2
exit 2
fi
if [ -d "$path" ]; then
find "$path" -maxdepth 1 -type f -name '*.jsonl' | sort
else
printf '%s\n' "$path"
fi
}

sources=$(
for arg in "$@"; do
resolve_source "$arg"
done
)

# ---- per-session metrics ----------------------------------------------
# $1 = file path -> stdout: one TSV line "calls out in cread ccreate final_ctx peak_ctx".
# jq confirms/parses each line and extracts the 4 usage fields (skipping a
# line that isn't valid JSON, or lacks .message.usage, rather than failing);
# awk does the integer aggregation (sums, running max, last-line capture).
session_metrics() {
"$JQ" -R -r '
(fromjson? // empty) as $obj
| ($obj.message.usage? // empty) as $u
| [($u.input_tokens // 0), ($u.cache_read_input_tokens // 0),
($u.cache_creation_input_tokens // 0), ($u.output_tokens // 0)]
| @tsv
' "$1" | awk -F'\t' '
{
in_s = $1 + 0; cr = $2 + 0; cc = $3 + 0; out = $4 + 0
ctx = in_s + cr + cc
calls++; out_sum += out; in_sum += in_s; cr_sum += cr; cc_sum += cc
final_ctx = ctx
if (ctx > peak_ctx) peak_ctx = ctx
}
END {
printf "%d\t%d\t%d\t%d\t%d\t%d\t%d\n", calls + 0, out_sum + 0, in_sum + 0, cr_sum + 0, cc_sum + 0, final_ctx + 0, peak_ctx + 0
}
'
}

# $1 = numerator $2 = denominator -> "N.N" (one decimal; "0.0" when denom is 0)
pct1() {
"$JQ" -nr --argjson n "$1" --argjson d "$2" '
if $d == 0 then "0.0"
else
(($n * 1000 / $d) | round) as $tenths
| ($tenths / 10 | floor) as $whole
| ($tenths - ($whole * 10)) as $frac
| "\($whole).\($frac)"
end
'
}

printf 'SESSION\tCALLS\tOUT\tHIT%%\tFINAL_CTX\tPEAK_CTX\n'

calls_total=0; out_total=0; in_total=0; cr_total=0; cc_total=0
final_max=0; peak_max=0
warnings=""

if [ -n "$sources" ]; then
while IFS= read -r file; do
[ -n "$file" ] || continue
name=$(basename "$file")
case "$name" in *.jsonl) name="${name%.jsonl}" ;; esac

metrics=$(session_metrics "$file")
calls=$(printf '%s' "$metrics" | cut -f1)
out=$(printf '%s' "$metrics" | cut -f2)
in_s=$(printf '%s' "$metrics" | cut -f3)
cr=$(printf '%s' "$metrics" | cut -f4)
cc=$(printf '%s' "$metrics" | cut -f5)
final_ctx=$(printf '%s' "$metrics" | cut -f6)
peak_ctx=$(printf '%s' "$metrics" | cut -f7)

hit=$(pct1 "$cr" "$((in_s + cr + cc))")
printf '%s\t%d\t%d\t%s\t%d\t%d\n' "$name" "$calls" "$out" "$hit" "$final_ctx" "$peak_ctx"

calls_total=$((calls_total + calls))
out_total=$((out_total + out))
in_total=$((in_total + in_s))
cr_total=$((cr_total + cr))
cc_total=$((cc_total + cc))
[ "$final_ctx" -gt "$final_max" ] && final_max=$final_ctx
[ "$peak_ctx" -gt "$peak_max" ] && peak_max=$peak_ctx

if [ "$peak_ctx" -ge "$warn_threshold" ]; then
warnings="${warnings}warn: $name peak context $peak_ctx >= $warn_threshold — consider phase-boundary /compact or delegating large reads
"
fi
done <<SOURCES_EOF
$sources
SOURCES_EOF
fi

hit_total=$(pct1 "$cr_total" "$((in_total + cr_total + cc_total))")
printf 'TOTAL\t%d\t%d\t%s\t%d\t%d\n' "$calls_total" "$out_total" "$hit_total" "$final_max" "$peak_max"

if [ -n "$warnings" ]; then
printf '%s' "$warnings" >&2
fi

exit 0
3 changes: 2 additions & 1 deletion skills/orchestrate/templates/brief.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,8 @@ specific tags below as authority.
inherits them: knowledge=<tool|default>, tacit=<tool|default>, plan=<tool|default> -->
<tools_guidance>{e.g. docs/specs to read, how to explore; DB read-only if any; resolved roles — knowledge/tacit/plan}</tools_guidance>

<constraints>{local rules; surgical changes only on shared files}</constraints>
<constraints>{local rules; surgical changes only on shared files}
token hygiene: bound tool output (tail/head, ranged reads); delegate visual checks to a subagent — see wiki/infrastructure/agent-orchestration/session-context-token-budget.md</constraints>

<!-- "what done looks like" — verifiable -->
<definition_of_done>
Expand Down
8 changes: 8 additions & 0 deletions skills/orchestrate/templates/session-prompt.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,3 +211,11 @@ and report exactly once:
[3] Forbidden — do not call any agent by name other than `test-quality-auditor`.
Others (e.g. a code-reviewer) may not exist in the user's environment and will
fail silently.

[4] Token hygiene — bound tool output before it reaches your context: pipe long
command output through `tail`/`head`, read big files by range
(offset/limit) instead of in full, and prefer a filename-only search mode
over one that returns full matched content. Delegate any screenshot or
visual verification to a subagent that returns a text verdict — an image
`Read` directly into this session is re-billed in full on every later turn.
See wiki/infrastructure/agent-orchestration/session-context-token-budget.md.
Empty file.
2 changes: 2 additions & 0 deletions tests/fixtures/token-report/edge/zero-usage.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"type":"summary"}
{"message":{"role":"assistant","content":"hi"}}
5 changes: 5 additions & 0 deletions tests/fixtures/token-report/session-a.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{"message":{"usage":{"input_tokens":100,"cache_read_input_tokens":0,"cache_creation_input_tokens":900,"output_tokens":50}}}
{"message":{"usage":{"input_tokens":100,"cache_read_input_tokens":800,"cache_creation_input_tokens":100,"output_tokens":150}}}
{"type":"summary"}
not valid json at all
{"message":{"usage":{"input_tokens":200,"cache_read_input_tokens":1800,"cache_creation_input_tokens":0,"output_tokens":300}}}
2 changes: 2 additions & 0 deletions tests/fixtures/token-report/session-b.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"message":{"usage":{"input_tokens":0,"cache_read_input_tokens":0,"cache_creation_input_tokens":1000,"output_tokens":100}}}
{"message":{"usage":{"input_tokens":0,"cache_read_input_tokens":1000,"cache_creation_input_tokens":0,"output_tokens":100}}}
81 changes: 81 additions & 0 deletions tests/orchestrate-token-budget.bats
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
#!/usr/bin/env bats
# Doc gates for t2-skill-token-directives: SKILL.md and the brief template
# must direct token-efficient coordinator/worker behavior, each citing the
# governing wiki page
# (wiki/infrastructure/agent-orchestration/session-context-token-budget.md).
# session-prompt.md's item [4] is already pinned by the byte-level cksum
# guard in tests/send-prompt.bats, so it is not duplicated here.
#
# Each gate is paired with a negative control (checks-that-cannot-pass,
# wiki/testing/quality/checks-that-cannot-pass.md): a fixture with the
# asserted span stripped, shown to fail the same check.

setup() {
REPO_ROOT="${BATS_TEST_DIRNAME}/.."
SKILL="${REPO_ROOT}/skills/orchestrate/SKILL.md"
BRIEF="${REPO_ROOT}/skills/orchestrate/templates/brief.md"
}

# Collapses embedded newlines to a single space so a substring assertion
# survives prose hard-wrapped across physical lines (same technique as
# tests/orchestrate-teardown-contract.bats).
normalize_ws() {
printf '%s' "$1" | tr '\n' ' ' | tr -s ' '
}

flat_skill() {
normalize_ws "$(cat "$SKILL")"
}

WIKI_SLUG='wiki/infrastructure/agent-orchestration/session-context-token-budget.md'

# --- doc-gate 1: Coordinator token budget section (D1) ---------------------

@test "doc-gate: Coordinator token budget section exists and cites the wiki page" {
flat="$(flat_skill)"
printf '%s' "$flat" | grep -qF "## Coordinator token budget"
printf '%s' "$flat" | grep -qF "$WIKI_SLUG"
printf '%s' "$flat" | grep -qF "delegate the visual check to a subagent"
printf '%s' "$flat" | grep -qF "/compact"
}

@test "doc-gate can fail: a fixture without the Coordinator token budget section does not match" {
fixture="${BATS_TEST_TMPDIR}/no-token-budget.md"
grep -v 'Coordinator token budget' "$SKILL" > "$fixture"
flat="$(normalize_ws "$(cat "$fixture")")"
count="$(printf '%s' "$flat" | grep -cF "## Coordinator token budget" || true)"
[ "$count" -eq 0 ]
}

# --- doc-gate 2: teardown token audit step (D2) -----------------------------

@test "doc-gate: End-of-run contract runs token-report.sh after teardown, citing the wiki page" {
flat="$(flat_skill)"
printf '%s' "$flat" | grep -qF "token-report.sh"
printf '%s' "$flat" | grep -qF -- "--cwd <root> ."
printf '%s' "$flat" | grep -qF -- "--cwd"
printf '%s' "$flat" | grep -qF "$WIKI_SLUG"
printf '%s' "$flat" | grep -qF "Exit 0 is the report, exit 2 a usage error, exit 4 a malformed"
}

@test "doc-gate can fail: a fixture without the token-report teardown step does not match" {
fixture="${BATS_TEST_TMPDIR}/no-token-report.md"
grep -v 'token-report.sh' "$SKILL" > "$fixture"
flat="$(normalize_ws "$(cat "$fixture")")"
count="$(printf '%s' "$flat" | grep -cF "token-report.sh" || true)"
[ "$count" -eq 0 ]
}

# --- doc-gate 3: brief.md token-hygiene constraint (D4) --------------------

@test "doc-gate: brief.md constraints carry the token-hygiene line citing the wiki page" {
grep -qF 'token hygiene' "$BRIEF"
grep -qF "$WIKI_SLUG" "$BRIEF"
}

@test "doc-gate can fail: a fixture without the token-hygiene line does not match" {
fixture="${BATS_TEST_TMPDIR}/brief-no-hygiene.md"
grep -v 'token hygiene' "$BRIEF" > "$fixture"
count="$(grep -cF 'token hygiene' "$fixture" || true)"
[ "$count" -eq 0 ]
}
Loading
Loading