perf(session-flow): run the description trim pilot and measure skill visibility #11954

Activity

1. kyle-sexton

   added

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

   status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.A maintainer confirmed the item is real and in scope; ready to be worked.

   host-runNeeds an agent run or builder session on a host; not a person or the issue lane.Needs an agent run or builder session on a host; not a person or the issue lane.

   on Nov 1, 2026
2. kyle-sexton

   added the

   Task

   issue type on Nov 1, 2026
3. kyle-sexton

   added a parent issue on Nov 1, 2026

   • skill-quality: skill authoring, quality and catalog #7582
4. kyle-sexton

   marked this as blocking 3 issues on Nov 1, 2026

   • fix(skill-quality): track unquoted triggers in check 3 and drop check 12's single-quote rule #11965
   • feat(skill-quality): read the description target and warn thresholds from settings #11971
   • marketplace: pstack recheck and listing budget #7588
5. kyle-sexton commented on Nov 1, 2026

   

   kyle-sexton

   on Nov 1, 2026

   MemberAuthor

   More actions

   Claimed. Backlog HITL host-run worker, 2026-11-01. Running the session-flow description trim pilot: live invocation probes (claude plugin eval, Skill-fired graders) on current versus trimmed descriptions, plus the skill-visibility listing measurement. Branch perf/11954-session-flow-trim-pilot. The coordinator posts the result.
6. kyle-sexton commented on Nov 1, 2026

   

   kyle-sexton

   on Nov 1, 2026

   MemberAuthor

   More actions

   Probe result: Host run. Ran the session-flow description trim pilot as live invocation probes (claude plugin eval) on the current and the trimmed descriptions, and took the skill-visibility listing measurement on both fleets.

   Verdict: the trim clears non-inferiority at the 5-point margin (Q185).

   • Pooled over all 187 paired should-trigger probes, the trigger rate went from 0.979 (183/187, baseline) to 0.984 (184/187, trim). Delta +0.005.
   • The 95% Newcombe paired interval (method 10) is [-0.026, +0.038], and its lower bound is above -0.05. A Wald interval gives [-0.018, +0.029].
   • Guard: the false-trigger rate on the 119 should-not-trigger probes went from 0.034 (4/119) to 0.042 (5/119). Delta +0.008, Newcombe [-0.033, +0.053].

   What ran

   • Claude Code 2.1.316 on the subscription login. Model claude-opus-5-5 was pinned in both arms, and every trace confirms it.
   • The trim covers 16 session-flow descriptions, shortened to the skill-quality: skill authoring, quality and catalog #7582 description standard (D82, Q183): one what-sentence, Use when with at most three quoted triggers, aiming for 250 characters, none over 419.
     • Total length went from 7,877 to 4,153 characters. Each description is now 240 to 282 characters, where before they ran 337 to 670.
     • Hook terms were chosen from the train probes only.
     • check (174 characters), setup and tidy-work are unchanged. setup and tidy-work are user-invoked and have no probes.
     • check-skill.sh plugins/session-flow/skills passed with 0 errors. Check 3 gave advisory "dropped trigger" warnings, which are expected with this standard.
   • Probes come from the plugin's existing plugins/session-flow/probes/ (17 files, 18 queries each, 306 in all).
     • Cases were generated with measure-invocation.sh emit-plugin-eval --runs 1.
     • Each case ran once per arm: claude plugin eval <arm>/session-flow --ablation none --keep-temp -j 2, at most two suites at once, each in a capped scope.
     • The baseline arm is plugins/session-flow copied from origin/main 8eca2165df. The trim arm is the same copy with only the 16 SKILL.md description lines changed.
   • Three passes, each run on both arms:
     1. 85 validation positives, with --disallow-tools Read Glob Grep.
     2. 102 train positives, with --disallow-tools Read Glob Grep Agent.
     3. 119 negatives (train and validation), with --disallow-tools Read Glob Grep Agent.
   • run-validity.py gave VALID for all six suites: every trace resolves, there were no run errors, no permission denials, and one model throughout.
     • The validation pass warns that Agent/Task stayed callable. Some kept traces in both arms do call them.
     • The train and negatives passes removed Agent, and pass the probe-tools check.

   Scores (1 run per case)

   Set	n	Baseline	Trim	Paired 2x2 (both / base only / trim only / neither)
   Validation positives	85	83 (0.976)	83 (0.976)	81 / 2 / 2 / 0
   Train positives	102	100 (0.980)	101 (0.990)	100 / 0 / 1 / 1
   All positives	187	183 (0.979)	184 (0.984)	181 / 2 / 3 / 1
   Negatives, false triggers	119	4 (0.034)	5 (0.042)	3 / 1 / 2 / 113

   • Changed positives:
     • Trim gained continue-in-background val-01, recall val-01 and running-retro train-05.
     • Trim lost continue-in-background val-04 and recommend-skills val-04.
   • Changed negatives:
     • The baseline alone false-triggered on audit-sessions train-03 and check val-03.
     • The trim alone false-triggered on orient train-03.
   • The lexical listing-overlap floor is not a model rate. It counts words shared with the listing, so a shorter description scores lower by construction:
     • Validation: positives fell from 51/85 to 36/85.
     • Train: positives fell from 64/102 to 49/102.
     • The live model rate did not fall.

   Limits on this result

   • There was 1 run per case, so per-case sampling noise is not averaged out. The paired design across 187 probes carries the decision.
   • validate gives 35 WARNs where a positive probe copies listing words. That favours the baseline wording, so it biases against the trim.
   • A first full-positives attempt was discarded. 184 of 187 baseline runs failed with "OAuth token revoked" while the trim arm kept running. The CLI was signed in when checked afterwards, and none of the runs reported here had an error. Cause unverified: possibly an account rotation mid-run.

   Skill-visibility measurement (gates Q183-Q187)

   The run was audit_skill_visibility.py --plugins-root over the whole 93-plugin checkout fleet (386 skills), base versus trim. Only session-flow differs between the two. The fraction is 0.07, read from .claude/settings.json.

   Band (window/bytes per token)	Budget	Demand base → trim	Overflow base → trim	Starved base → trim
   200k/4	56,000	155,570 → 151,846	107,856 → 104,132	221 → 216
   200k/3	42,000	155,570 → 151,846	121,856 → 118,132	245 → 242
   1M/4	280,000	fits	0	0 (fit-unconfirmed)
   1M/3	210,000	fits	0	0 (fit-unconfirmed)

   • Session-flow's own competing demand fell from 8,051 to 4,327 characters (-46%).
   • Session-flow rows marked likely-starved: 8 → 8 at 200k/4, and 12 → 11 at 200k/3.
   • At 200k windows the fleet still overflows the 0.07 budget about threefold. Trimming one plugin removes about 3% of fleet demand. That figure is the input to marketplace: pstack recheck and listing budget #7588 / feat(context-budget): keep plugin skill descriptions visible when the skill listing overflows #6458, which stay out of scope here.

   skillListingBudgetFraction: unchanged at 0.07. It is at .claude/settings.json:61 on origin/main 8eca2165df and on the trim head (Q187). This slice does not touch it.

   Branch: perf/11954-session-flow-trim-pilot, head ba4f5ab39b933a875bc4d1f5c21a01af6345f67d, pushed.

   • It holds one commit, 16 SKILL.md description lines plus .changes/session-flow/perf-11954-description-trim-pilot-7c3e91a4.md.
   • scripts/check-affected.sh --tests on the committed head: 14 passed, 2 skipped.
   • The coordinator opens the PR.

   Target labels: add none; remove host-run.
7. kyle-sexton

   removed

   host-runNeeds an agent run or builder session on a host; not a person or the issue lane.Needs an agent run or builder session on a host; not a person or the issue lane.

   on Nov 1, 2026
8. melodic-automation-lanes

   marked this as blocking on Nov 1, 2026

   • perf(context-budget): measure the fleet listing fraction on Opus 5.5 and Fable 5.1 and set skillListingBudgetFraction #12044
9. kyle-sexton

   closed this as completedin #12021on Nov 1, 2026