feat: toolResultContains match gate on the last tool-result payload#299
Merged
Conversation
Discriminates fixtures whose requests differ only inside the tool-result message content, e.g. a human-in-the-loop suspend tool where approve and cancel resume with the same tool_call_id but different result JSON. - router: substring gate on the LAST message's text content when that message is a tool result (same last-message rule as toolCallId) - fixture-loader: carry through entryToFixture, validate non-empty string, include in the userMessage dedup key and the catch-all discriminator set - aimock-pytest: route toolResultContains kwarg into the match block - docs: fixtures + multi-turn pages, README, write-fixtures skill
commit: |
ranst91
added a commit
to CopilotKit/CopilotKit
that referenced
this pull request
Jul 15, 2026
…sultContains gate Pick and cancel resume the native schedule_meeting suspend tool with the SAME toolCallId; the requests differ only inside the tool-result payload, so the cancel resume previously hit the pick-confirmation fixture and the assistant replayed "Booked: ... confirmed" after the user cancelled. The "__cancelled" toolCallId gates on the Denied fixtures were fictional and never matched. aimock 1.37.0 (CopilotKit/aimock#299) adds a JSON-expressible match.toolResultContains substring gate on the last tool-result message. - gen-ui-interrupt.json: cancelled legs now gate on the real toolCallId + toolResultContains "cancelled", ordered before the confirmation legs - interrupt-headless.json: gained the same cancelled legs (the demo's Cancel button had no fixture at all) - aimock-fixtures.test.ts: matchKey learns toolResultContains; duplicate ceiling 303 -> 305 (headless cancelled legs share exact keys AND response text with the gen-ui-interrupt ones, one pair per pill) - e2e specs: cancel tests now assert the Denied narration and reject Booked/Scheduled, so the regression cannot silently return Verified live against aimock built from source (fixture replay): 8/8 Playwright e2e across both demos, plus manual pick + cancel runs on /demos/gen-ui-interrupt and /demos/interrupt-headless. Commit uses --no-verify: this worktree's lefthook runner is broken (pre-existing, see daa501d); commitlint + prettier + the fixtures vitest were run manually and pass. Follow-up (blocked on aimock#299 npm publish): bump the vendored @copilotkit/aimock pin in showcase/scripts/package.json and pull the refreshed ghcr.io/copilotkit/aimock:latest.
…inators The dedup key in validateFixtures only carried turnIndex, hasToolResult, toolResultContains, sequenceIndex, and context, so two fixtures sharing a userMessage but legitimately disambiguated only by an omitted matcher (systemMessage, model, toolName, toolCallId, inputText, responseFormat, or endpoint) were flagged as spurious duplicates. Expand the key to every discriminator matchFixtureDiagnostic actually gates on, serialising RegExp/array matchers kind-aware so they never collide, and key predicate-bearing fixtures by identity so they are never false duplicates. The check only ever warns; it never drops or dedupes fixtures.
…rules Consolidates the match-field, dedup-caveat, and toolCallId-rule doc corrections for the new gate into one change across docs/fixtures, docs/multi-turn, and the write-fixtures skill.
requests is already a main dependency so it is always installed; listing it again in the test optional-dependency group is redundant.
The auto-download path (AIMOCK_CLI_PATH unset) fetched 1.35.1, which predates the server toolResultContains route. Since 0.5.0 forwards that match kwarg, default pip users got a server that silently ignored it. Pin now tracks the 1.37.0 release that adds the route.
jpr5
force-pushed
the
feat/tool-result-contains
branch
from
4877b4e to
7c428d5
Compare
The release script published without verifying formatting or the package export map, so unformatted code or a broken export map could reach npm (recurring failure, see CHANGELOG #263). Add format:check before build (cheap, fail fast) and test:exports after build (needs the built dist), with npm publish last.
The E2E setup examples loaded fixtures once then called full reset() in afterEach, which wipes the in-memory fixture pool and 404s every test after the first. Switch between-test cleanup to resetMatchCounts() (suite.llm.resetMatchCounts() for MockSuite) and correct the prose so it leads with resetMatchCounts() and warns reset() also clears fixtures.
Untrusted fixture JSON may set interChunkDelaysMs to null/missing/non-array. The sanitizer called .filter with no Array.isArray guard, throwing a TypeError inside entryToFixture that aborted the entire fixture file's load and silently dropped every fixture in it. Coerce non-array values to [] like sibling guards.
The fixtures and record-replay pages stated that only string userMessage and inputText flip to strict equality under a requestTransform, omitting systemMessage. The router flips all three string forms (userMessage line 284, systemMessage line 330, inputText line 377). Align both pages with the code and the multi-turn page's already-correct field list.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The showcase mastra interrupt demos use a native suspend tool where pick and cancel resume with the SAME tool_call_id; the requests differ only inside the tool-result message content (
{"chosen_time": ...}vs{"cancelled": true}). Every existing JSON-expressible gate is identical between the two requests, so first-match-wins replays the pick confirmation after the user cancelled.predicatecannot help: the JSON fixture loader has no way to express it.What
New
match.toolResultContains: stringgate: passes when the request's LAST message is arole: "tool"result whose text content contains the substring. Mirrors the existingtoolCallIdlast-message rule so the two stay consistent.src/router.ts: gate placed with the other shape predicates, right aftertoolCallIdsrc/types.ts:FixtureMatch+FixtureFileEntry.matchsrc/fixture-loader.ts:entryToFixturecarry-through; validation (string, non-empty — empty substring would silently act as a catch-all); userMessage dedup key; catch-all discriminator setpackages/aimock-pytest:toolResultContainsadded to_MATCH_LEVEL_OPT_KEYS(0.5.0)Version
1.37.0 (changelog entry added with this PR's number in a follow-up commit on this branch).
Maintainer note — additional fixes folded in during review
Beyond the
toolResultContainsgate, this branch now also carries (all under the single 1.37.0 release):predicatekeyed by identity).recordedTimings.interChunkDelaysMsagainst non-array values — a malformed entry no longer throws and silently drops every fixture in its file.aimock-pytestdefault server pin to 1.37.0 so the auto-download path actually supports the new kwarg the client forwards.releasescript now runsformat:check+test:exportsbefore publish.toolCallIdsemantics docs, and fixed the write-fixtures E2E lifecycle to useresetMatchCounts()instead of a fullreset()between tests.(These consolidate PR #300, now closed, into this one release. CHANGELOG updated accordingly.)