Summary
Bring Copilot's testing → test-servers hand-off to 100% on both chain cases in .claude/skills/test-servers/evals/evals.json. Claude already makes both hand-offs 100% of the time. Copilot makes the integration-test one every time, but the pagination one only 3 times in 5.
Current state (v2.8.0 release smoke, Copilot CLI 1.0.88)
AGENT=copilot RUNS=5 npm run skills:eval -- test-servers on the v2.8.0 merge tree (#2456):
PASS 100% testing → test-servers Write an integration test that exercises tool listing end to end.
PASS 60% testing → test-servers Add end-to-end coverage for the tool-list pagination path.
That matches what #2402 recorded after the #2399 wording change (40% → 60%). A single full-suite pass (RUNS=1) hit both hand-offs, and all 71 first-move cases were at 100%.
How it misses
From #2402's recorded runs on the pagination prompt, the two failure shapes are:
- Loads
testing, then greps for an existing test, finds test-servers/configs/pagination-http.json, and copies from it without following the pointer. The pointer reads as conditional on whether the test uses a fixture, and the model answers that question by searching instead of loading the skill.
- Greps first and loads no skill at all, sometimes reaching
testing → test-servers late.
"End to end" in a prompt apparently isn't a strong enough cue for Copilot, while "integration test" is.
Levers to try
Measure each one separately (probe, then measure, per docs/skill-authoring.md):
- The
testing body: move the test-servers pointer to the top of the body, and make it unconditional for "end-to-end / integration coverage of an MCP operation" rather than conditional on a fixture question.
- The
testing description: name "end-to-end coverage" and "pagination" among its trigger situations, so the first link fires before any grep.
- The
test-servers description: add "end-to-end coverage of an MCP operation" as a situation. That can help Copilot reach it directly, but a first-move test-servers load is not what the chain case scores, so it only helps if the chain still goes through testing.
⚠️ Per AGENTS.md (Maintaining the skills), any description edit can lower other skills' trigger rates, so re-run the full eval under both agents after each change, not just these two cases.
Acceptance
Summary
Bring Copilot's
testing → test-servershand-off to 100% on both chain cases in.claude/skills/test-servers/evals/evals.json. Claude already makes both hand-offs 100% of the time. Copilot makes the integration-test one every time, but the pagination one only 3 times in 5.Current state (v2.8.0 release smoke, Copilot CLI 1.0.88)
AGENT=copilot RUNS=5 npm run skills:eval -- test-serverson the v2.8.0 merge tree (#2456):That matches what #2402 recorded after the #2399 wording change (40% → 60%). A single full-suite pass (
RUNS=1) hit both hand-offs, and all 71 first-move cases were at 100%.How it misses
From #2402's recorded runs on the pagination prompt, the two failure shapes are:
testing, then greps for an existing test, findstest-servers/configs/pagination-http.json, and copies from it without following the pointer. The pointer reads as conditional on whether the test uses a fixture, and the model answers that question by searching instead of loading the skill.testing → test-serverslate."End to end" in a prompt apparently isn't a strong enough cue for Copilot, while "integration test" is.
Levers to try
Measure each one separately (probe, then measure, per
docs/skill-authoring.md):testingbody: move thetest-serverspointer to the top of the body, and make it unconditional for "end-to-end / integration coverage of an MCP operation" rather than conditional on a fixture question.testingdescription: name "end-to-end coverage" and "pagination" among its trigger situations, so the first link fires before any grep.test-serversdescription: add "end-to-end coverage of an MCP operation" as a situation. That can help Copilot reach it directly, but a first-movetest-serversload is not what the chain case scores, so it only helps if the chain still goes throughtesting.AGENTS.md(Maintaining the skills), any description edit can lower other skills' trigger rates, so re-run the full eval under both agents after each change, not just these two cases.Acceptance
AGENT=copilot RUNS=5 npm run skills:eval -- test-serversreports 100% on both hand-off cases, on two separate runs (at n=5, one run at 100% could be luck)AGENT=claude(the default) still reports 100% on both hand-offsnpm run verify:skillspasses, and the listing budget stays within 4,000 characterstest-serversbody as a caller arriving from the pagination prompt would, and confirm it answers that task, not just that it loaded