agentHost: pin tool-search behavior in prompt snapshots - #333689
Draft
Bhavya U (bhavyaus) wants to merge 4 commits into
Draft
agentHost: pin tool-search behavior in prompt snapshots#333689Bhavya U (bhavyaus) wants to merge 4 commits into
Bhavya U (bhavyaus) wants to merge 4 commits into
Conversation
Contributor
There was a problem hiding this comment.
Copilot review overview
Review tier: Balanced
Findings: 2
New issues introduced by this change (3)
| Severity | Finding |
|---|---|
 |
src/vs/platform/agentHost/test/node/e2e/harness/agentHostE2ETestHarness.ts — The E2E architecture requires this suite to remain external to the Agent Host and explicitly… |
|
src/vs/platform/agentHost/test/node/e2e/providers/copilotPromptsE2E.integrationTest.ts — These expectations come directly from the implementation under test, contrary to the E2E suite's… |
 |
src/vs/platform/agentHost/test/node/e2e/providers/copilotPromptsE2E.integrationTest.ts — The current predicate returns true for gpt-5.6-sol, gpt-5.6-luna, and gpt-5.6-terra, and… |
What changed in this PR
Updates Agent Host prompt snapshots to exercise and pin tool-search gating.
Changes:
- Adds reusable E2E helpers for root configuration and active-client tools.
- Asserts tool-search guidance and carrier behavior.
- Regenerates all 19 model prompt snapshots.
| File | Description |
|---|---|
copilotCoverageSuite.ts |
Reuses canonical harness helpers. |
copilotPromptsE2E.integrationTest.ts |
Enables and asserts tool-search behavior. |
agentHostE2ETestHarness.ts |
Adds configuration and client-tool helpers. |
toolInstructions.ts |
Exports the tool-search guidance text. |
…gpt-5.prompt.md |
Adds canonical client tools. |
…gpt-5-mini.prompt.md |
Adds canonical client tools. |
…gpt-5-codex.prompt.md |
Adds canonical client tools. |
…gpt-5_1.prompt.md |
Adds canonical client tools. |
…gpt-5_1-codex.prompt.md |
Adds canonical client tools. |
…gpt-5_1-codex-mini.prompt.md |
Adds canonical client tools. |
…gpt-5_6-sol.prompt.md |
Adds client tools and guidance. |
…gpt-5_6-luna.prompt.md |
Adds client tools and guidance. |
…gpt-5_6-terra.prompt.md |
Adds client tools and guidance. |
…gemini-2_0-flash.prompt.md |
Adds canonical client tools. |
…claude-haiku-4_5.prompt.md |
Adds client tools and guidance. |
…claude-sonnet-4_5.prompt.md |
Adds client tools and guidance. |
…claude-sonnet-4_6.prompt.md |
Adds client tools and guidance. |
…claude-sonnet-5.prompt.md |
Adds client tools and guidance. |
…claude-opus-4_5.prompt.md |
Adds client tools and guidance. |
…claude-opus-4_6.prompt.md |
Adds client tools and guidance. |
…claude-opus-4_7.prompt.md |
Adds client tools and guidance. |
…claude-opus-4_8.prompt.md |
Adds client tools and guidance. |
…claude-opus-5.prompt.md |
Adds client tools and guidance. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| } from '../../../../common/state/sessionState.js'; | ||
| import { SEMANTIC_SEARCH_TOOL_NAME } from '../../../../common/semanticSearchConstants.js'; | ||
| import { CLIENT_TOOL_SEARCH_REFERENCE_NAME } from '../../../../common/toolSearchConstants.js'; | ||
| import { NON_DEFERRED_CLIENT_TOOL_NAMES } from '../../../../node/copilot/toolSearchDeferral.js'; |
Comment on lines
+40
to
+41
| import { agentHostModelSupportsToolSearch, RUNTIME_TOOL_SEARCH_TOOL_NAME } from '../../../../node/copilot/toolSearchDeferral.js'; | ||
| import { COPILOT_AGENT_HOST_TOOL_SEARCH_TOOL_INSTRUCTION } from '../../../../node/copilot/prompts/toolInstructions.js'; |
| // green. The CLI applies a further gate of its own before exposing | ||
| // `tool_search_tool` (unsatisfied under replay), so only the | ||
| // host-authored guidance line is pinned in both directions. | ||
| const supportsToolSearch = agentHostModelSupportsToolSearch(model); |
The prompt snapshot suite never advertised the `toolSearch` client tool, so `toolSearchActive` was always false and the snapshots pinned nothing about tool search. Re-enabling GPT tool search in #333657 produced a zero-line snapshot diff, which is what surfaced the gap. Give sessions the harness's canonical client-tool profile so the host's client-tool-gated launch path is actually exercised, and assert on the host-authored guidance line in both directions using the production `agentHostModelSupportsToolSearch` predicate rather than a hand-copied model list. The CLI applies a second gate before exposing `tool_search_tool` that is unsatisfied under replay, so the carrier itself cannot be asserted here. `toolSearchToolWithoutHostGate` still catches a carrier leaking onto a model the host gate excludes. Extracts `setRootConfigValues`, `canonicalClientTools` and `registerCanonicalActiveClient` into the harness, de-duplicating the inline active-client setup in the tool-search coverage tests. Also hoists the tool-search guidance string in `prompts/toolInstructions.ts` to an exported constant so the test asserts against it rather than duplicating 318 characters of prompt text. The template literal had no interpolation, so this is a pure extraction. All 19 model snapshots regenerate. 12 gain the guidance line (9 Claude plus gpt-5.6-sol/luna/terra, which `agentHostModelSupportsToolSearch` covers via `isGpt56Model`); the other 7 gain only the canonical client tools. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Bhavya U (bhavyaus)
force-pushed
the
dev/bhavyau/tool-search-prompt-snapshot
branch
from
August 31, 2026 23:42
15a630a to
ed8f780
Compare
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep the prompt harness focused on the VS Code Copilot extension's toolSearch AHP reference rather than asserting on the SDK runtime's native carrier. Remove the duplicated investigation note from the repository. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.