From 1d873cc42000d8ec87ea8743b99cafd85020b97b Mon Sep 17 00:00:00 2001 From: ran Date: Wed, 15 Jul 2026 12:40:56 -0700 Subject: [PATCH 01/10] feat: add toolResultContains match gate on the last tool-result payload Discriminates fixtures whose requests differ only inside the tool-result message content, e.g. a human-in-the-loop suspend tool where approve and cancel resume with the same tool_call_id but different result JSON. - router: substring gate on the LAST message's text content when that message is a tool result (same last-message rule as toolCallId) - fixture-loader: carry through entryToFixture, validate non-empty string, include in the userMessage dedup key and the catch-all discriminator set - aimock-pytest: route toolResultContains kwarg into the match block - docs: fixtures + multi-turn pages, README, write-fixtures skill --- README.md | 2 +- docs/fixtures/index.html | 10 ++ docs/multi-turn/index.html | 10 ++ package.json | 2 +- packages/aimock-pytest/pyproject.toml | 2 +- .../src/aimock_pytest/_server.py | 1 + skills/write-fixtures/SKILL.md | 57 +++++---- src/__tests__/fixture-loader.test.ts | 49 +++++++ src/__tests__/router.test.ts | 121 ++++++++++++++++++ src/fixture-loader.ts | 29 ++++- src/router.ts | 13 ++ src/types.ts | 13 ++ 12 files changed, 277 insertions(+), 32 deletions(-) diff --git a/README.md b/README.md index 687ca280..4203bd44 100644 --- a/README.md +++ b/README.md @@ -50,7 +50,7 @@ Run them all on one port with `npx @copilotkit/aimock --config aimock.json`, or - **[Record & Replay](https://aimock.copilotkit.dev/record-replay)** — Proxy real APIs, save as fixtures, replay deterministically forever - **Timing-aware recording and replay** — Recorded fixtures capture per-frame arrival timestamps; replay uses recorded timings for approximate timing reproduction based on recorded TTFT and inter-frame cadence (replay chunk count may differ from recording — TTFT and average pace are preserved, not per-token fidelity) with configurable `--replay-speed` multiplier -- **[Multi-turn Conversations](https://aimock.copilotkit.dev/multi-turn)** — Record and replay multi-turn traces with tool rounds; match distinct turns via `turnIndex`, `hasToolResult`, `toolCallId`, `sequenceIndex`, `systemMessage` (gate on host-supplied agent context), or custom predicates +- **[Multi-turn Conversations](https://aimock.copilotkit.dev/multi-turn)** — Record and replay multi-turn traces with tool rounds; match distinct turns via `turnIndex`, `hasToolResult`, `toolCallId`, `toolResultContains` (gate on the tool-result payload), `sequenceIndex`, `systemMessage` (gate on host-supplied agent context), or custom predicates - **[12 providers across 14 API surfaces](https://aimock.copilotkit.dev/docs)** — OpenAI Chat, OpenAI Responses, OpenAI Realtime (GA + Beta shim), Claude, Gemini (REST + embedContent), Gemini Live, Gemini Interactions, Azure, Bedrock, Vertex AI, Ollama (chat + embeddings), Cohere (chat + embed), ElevenLabs TTS — full streaming support - **Multimedia APIs** — [image generation](https://aimock.copilotkit.dev/images) (DALL-E, Imagen), [image editing](https://aimock.copilotkit.dev/images) (/v1/images/edits), [text-to-speech](https://aimock.copilotkit.dev/speech) (OpenAI + ElevenLabs), [audio transcription](https://aimock.copilotkit.dev/transcription), [audio translation](https://aimock.copilotkit.dev/transcription) (/v1/audio/translations), [video generation](https://aimock.copilotkit.dev/video), [OpenRouter video generation](https://aimock.copilotkit.dev/openrouter-video) (/api/v1/videos with async job lifecycle), [Google Veo video generation](https://aimock.copilotkit.dev/veo-video) (:predictLongRunning + /v1beta/operations async lifecycle), [Grok Imagine video generation](https://aimock.copilotkit.dev/grok-video) (/v1/videos/generations with async job lifecycle), [fal.ai](https://aimock.copilotkit.dev/fal-ai) (image / video / audio with queue lifecycle) - **[MCP](https://aimock.copilotkit.dev/mcp-mock) / [A2A](https://aimock.copilotkit.dev/a2a-mock) / [AG-UI](https://aimock.copilotkit.dev/agui-mock) / [Vector](https://aimock.copilotkit.dev/vector-mock)** — Mock every protocol your AI agents use diff --git a/docs/fixtures/index.html b/docs/fixtures/index.html index 67a0b2d9..7852e0e4 100644 --- a/docs/fixtures/index.html +++ b/docs/fixtures/index.html @@ -110,6 +110,16 @@

Match Fields

The onToolResult(id, response) helper is sugar over this field + + toolResultContains + string + + Substring match on the text content of the last message when it is a + role: "tool" result. Discriminates fixtures whose requests differ only + inside the tool-result payload (e.g. approve vs cancel resuming with the same + tool_call_id). See Multi-Turn Conversations + + toolName string diff --git a/docs/multi-turn/index.html b/docs/multi-turn/index.html index 3b56a788..3982e73c 100644 --- a/docs/multi-turn/index.html +++ b/docs/multi-turn/index.html @@ -72,6 +72,16 @@

How matching works across turns

distinguish the turn that requests a tool from the turn that follows up on a tool result. +
  • + toolResultContains (v1.37.0) is a substring match against the + text content of the last tool message (same last-message rule as + toolCallId). Use it when two resume paths share the same + tool_call_id and differ only inside the tool-result payload — e.g. a + human-in-the-loop tool where approve resumes with + {"chosen_time": …} and cancel resumes with + {"cancelled": true}. Gate the cancel fixture on + "toolResultContains": "cancelled" and order it before the approve leg. +

    • A request carrying a 20-message history still only matches on its last user message (and, diff --git a/package.json b/package.json index 8fd650c2..87b8165d 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@copilotkit/aimock", - "version": "1.36.1", + "version": "1.37.0", "description": "Mock infrastructure for AI application testing — LLM APIs, image generation, image editing, text-to-speech, transcription, audio translation, audio generation, video generation, embeddings, MCP tools, A2A agents, AG-UI event streams, vector databases, search, rerank, and moderation. One package, one port, zero dependencies.", "license": "MIT", "keywords": [ diff --git a/packages/aimock-pytest/pyproject.toml b/packages/aimock-pytest/pyproject.toml index 09e0894f..39666a5c 100644 --- a/packages/aimock-pytest/pyproject.toml +++ b/packages/aimock-pytest/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "aimock-pytest" -version = "0.4.0" +version = "0.5.0" description = "pytest fixtures for aimock — mock LLM APIs, multimedia, MCP, A2A, AG-UI, vector DBs, and more" readme = "README.md" requires-python = ">=3.10" diff --git a/packages/aimock-pytest/src/aimock_pytest/_server.py b/packages/aimock-pytest/src/aimock_pytest/_server.py index 946902de..07d98a09 100644 --- a/packages/aimock-pytest/src/aimock_pytest/_server.py +++ b/packages/aimock-pytest/src/aimock_pytest/_server.py @@ -182,6 +182,7 @@ def url(self) -> str: "systemMessage", "inputText", "toolCallId", + "toolResultContains", "toolName", "model", "responseFormat", diff --git a/skills/write-fixtures/SKILL.md b/skills/write-fixtures/SKILL.md index 8c3de1a4..72906c51 100644 --- a/skills/write-fixtures/SKILL.md +++ b/skills/write-fixtures/SKILL.md @@ -19,25 +19,26 @@ aimock is a zero-dependency mock infrastructure for AI apps. Fixture-driven. Mul ## Match Field Reference -| Field | Type | Matches Against | -| ---------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `userMessage` | `string` | Substring of last `role: "user"` message text | -| `userMessage` | `RegExp` | Pattern test on last `role: "user"` message text | -| `systemMessage` | `string` | Substring of the concatenated text of every `role: "system"` message in the request. Use to gate a fixture on host-supplied context (persona, agent-context entries) so changes to that context cause the fixture to fall through instead of returning a stale baked response | -| `systemMessage` | `string[]` | Array of substrings — ALL must be present in the joined system text (AND semantics). Use when the gate must combine multiple non-adjacent tokens whose serialisation order isn't stable | -| `systemMessage` | `RegExp` | Pattern test on the concatenated system-message text | -| `inputText` | `string` | Substring of embedding input text (concatenated if multiple inputs) | -| `inputText` | `RegExp` | Pattern test on embedding input text | -| `toolName` | `string` | Exact match on any tool in request's `tools[]` array (by `function.name`) | -| `toolCallId` | `string` | Exact match on `tool_call_id` of last `role: "tool"` message | -| `model` | `string` | Exact match on `req.model` | -| `model` | `RegExp` | Pattern test on `req.model` | -| `responseFormat` | `string` | Exact match on `req.response_format.type` (`"json_object"`, `"json_schema"`) | -| `sequenceIndex` | `number` | Matches only when this fixture's match count equals the given index (0-based) | -| `turnIndex` | `number` | Stateless conversation-depth matching. Counts `role: "assistant"` messages in the request; matches when that count equals the value. `turnIndex: 0` = first turn (no prior assistant messages). Use instead of `sequenceIndex` for shared/deployed instances where stateful counters break under concurrency | -| `hasToolResult` | `boolean` | Stateless tool-message presence matching. `true` matches when any `role: "tool"` message exists in the request; `false` matches when none exist. Provider-consistent across all aimock handlers (OpenAI, Claude, Gemini, Bedrock, Ollama, Cohere) | -| `endpoint` | `string` | Restrict to endpoint type: `"chat"`, `"image"`, `"speech"`, `"transcription"`, `"video"`, `"embedding"` | -| `predicate` | `(req: ChatCompletionRequest) => boolean` | Custom function — full access to request | +| Field | Type | Matches Against | +| -------------------- | ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `userMessage` | `string` | Substring of last `role: "user"` message text | +| `userMessage` | `RegExp` | Pattern test on last `role: "user"` message text | +| `systemMessage` | `string` | Substring of the concatenated text of every `role: "system"` message in the request. Use to gate a fixture on host-supplied context (persona, agent-context entries) so changes to that context cause the fixture to fall through instead of returning a stale baked response | +| `systemMessage` | `string[]` | Array of substrings — ALL must be present in the joined system text (AND semantics). Use when the gate must combine multiple non-adjacent tokens whose serialisation order isn't stable | +| `systemMessage` | `RegExp` | Pattern test on the concatenated system-message text | +| `inputText` | `string` | Substring of embedding input text (concatenated if multiple inputs) | +| `inputText` | `RegExp` | Pattern test on embedding input text | +| `toolName` | `string` | Exact match on any tool in request's `tools[]` array (by `function.name`) | +| `toolCallId` | `string` | Exact match on `tool_call_id` of last `role: "tool"` message | +| `toolResultContains` | `string` | Substring of the last tool message's text content, gated on that message being the request's LAST message (same rule as `toolCallId`). Discriminates resume paths that share a `tool_call_id` and differ only inside the tool-result payload (e.g. approve `{"chosen_time": …}` vs cancel `{"cancelled": true}`) | +| `model` | `string` | Exact match on `req.model` | +| `model` | `RegExp` | Pattern test on `req.model` | +| `responseFormat` | `string` | Exact match on `req.response_format.type` (`"json_object"`, `"json_schema"`) | +| `sequenceIndex` | `number` | Matches only when this fixture's match count equals the given index (0-based) | +| `turnIndex` | `number` | Stateless conversation-depth matching. Counts `role: "assistant"` messages in the request; matches when that count equals the value. `turnIndex: 0` = first turn (no prior assistant messages). Use instead of `sequenceIndex` for shared/deployed instances where stateful counters break under concurrency | +| `hasToolResult` | `boolean` | Stateless tool-message presence matching. `true` matches when any `role: "tool"` message exists in the request; `false` matches when none exist. Provider-consistent across all aimock handlers (OpenAI, Claude, Gemini, Bedrock, Ollama, Cohere) | +| `endpoint` | `string` | Restrict to endpoint type: `"chat"`, `"image"`, `"speech"`, `"transcription"`, `"video"`, `"embedding"` | +| `predicate` | `(req: ChatCompletionRequest) => boolean` | Custom function — full access to request | **AND logic**: all specified fields must match. Empty match `{}` = catch-all. @@ -45,12 +46,13 @@ Multi-part content (e.g., `[{type: "text", text: "hello"}]`) is automatically ex ### When to Use Each Multi-turn Matching Approach -| Approach | Stateless? | Best For | -| --------------- | ---------- | --------------------------------------------------------------------------------------------------------- | -| `turnIndex` | Yes | Shared/deployed instances; matches on conversation depth (count of assistant messages in request) | -| `hasToolResult` | Yes | Simplest option for 2-step tool flows — boolean: are there tool results in the request? | -| `sequenceIndex` | No | Single-client unit tests with repeated identical requests (server-side counter, breaks under concurrency) | -| `toolCallId` | Yes | Matching specific tool result IDs in the conversation history | +| Approach | Stateless? | Best For | +| -------------------- | ---------- | --------------------------------------------------------------------------------------------------------- | +| `turnIndex` | Yes | Shared/deployed instances; matches on conversation depth (count of assistant messages in request) | +| `hasToolResult` | Yes | Simplest option for 2-step tool flows — boolean: are there tool results in the request? | +| `sequenceIndex` | No | Single-client unit tests with repeated identical requests (server-side counter, breaks under concurrency) | +| `toolCallId` | Yes | Matching specific tool result IDs in the conversation history | +| `toolResultContains` | Yes | Same tool call id, different outcomes — match on the tool-result payload (approve vs cancel legs) | **Prefer stateless approaches** (`turnIndex`, `hasToolResult`) for shared aimock instances (deployed via Docker, used by multiple test runners). Use `sequenceIndex` only in isolated single-client unit tests where the counter won't be corrupted by concurrent requests. @@ -64,6 +66,11 @@ Multi-part content (e.g., `[{type: "text", text: "hello"}]`) is automatically ex // Same thing with hasToolResult (simpler for 2-step) {"match": {"userMessage": "trip to mars", "hasToolResult": false}, "response": {"toolCalls": [{"id": "call_001", "name": "generate_steps", "arguments": "{}"}]}} {"match": {"userMessage": "trip to mars", "hasToolResult": true}, "response": {"content": "Great choices!"}} + +// HITL suspend tool where approve and cancel resume with the SAME tool call id — +// discriminate on the tool-result payload; put the cancel leg first (first match wins) +{"match": {"toolCallId": "call_001", "toolResultContains": "\"cancelled\""}, "response": {"content": "No problem — nothing was booked."}} +{"match": {"toolCallId": "call_001"}, "response": {"content": "Booked: Monday 9:00 AM confirmed."}} ``` ## Response Types diff --git a/src/__tests__/fixture-loader.test.ts b/src/__tests__/fixture-loader.test.ts index ca4719c1..67c163d4 100644 --- a/src/__tests__/fixture-loader.test.ts +++ b/src/__tests__/fixture-loader.test.ts @@ -1233,6 +1233,34 @@ describe("validateFixtures", () => { expect(results.filter((r) => r.message.includes("hasToolResult"))).toHaveLength(0); }); + // --- match.toolResultContains type checks --- + + it("error: toolResultContains is a number", () => { + const fixtures = [ + makeFixture({ match: { userMessage: "test", toolResultContains: 42 as never } }), + ]; + const results = validateFixtures(fixtures); + expect( + results.some((r) => r.severity === "error" && r.message.includes("toolResultContains")), + ).toBe(true); + }); + + it("error: toolResultContains is an empty string", () => { + const fixtures = [makeFixture({ match: { userMessage: "test", toolResultContains: "" } })]; + const results = validateFixtures(fixtures); + expect( + results.some((r) => r.severity === "error" && r.message.includes("toolResultContains")), + ).toBe(true); + }); + + it("no error: toolResultContains is a non-empty string", () => { + const fixtures = [ + makeFixture({ match: { userMessage: "test", toolResultContains: "cancelled" } }), + ]; + const results = validateFixtures(fixtures); + expect(results.filter((r) => r.message.includes("toolResultContains"))).toHaveLength(0); + }); + // --- match.systemMessage type checks --- it("error: systemMessage is a number", () => { @@ -1348,6 +1376,18 @@ describe("validateFixtures", () => { expect(duplicateWarnings).toHaveLength(0); }); + it("no warning: same userMessage but different toolResultContains", () => { + const fixtures = [ + makeFixture({ match: { userMessage: "hello", toolResultContains: "cancelled" } }), + makeFixture({ match: { userMessage: "hello", toolResultContains: "chosen_" } }), + ]; + const results = validateFixtures(fixtures); + const duplicateWarnings = results.filter( + (r) => r.severity === "warning" && r.message.includes("duplicate"), + ); + expect(duplicateWarnings).toHaveLength(0); + }); + it("no warning: same userMessage but different sequenceIndex", () => { const fixtures = [ makeFixture({ match: { userMessage: "hello", sequenceIndex: 0 } }), @@ -1868,6 +1908,15 @@ describe("auto-stringify JSON objects in fixture entries", () => { expect(fixture.match.systemMessage).toBe("name=Atai"); }); + it("passes toolResultContains through entryToFixture", () => { + const entry: FixtureFileEntry = { + match: { toolCallId: "call_x", toolResultContains: '"cancelled"' }, + response: { content: "ok" }, + }; + const fixture = entryToFixture(entry); + expect(fixture.match.toolResultContains).toBe('"cancelled"'); + }); + it("stringifies nested objects in arguments", () => { const entry: FixtureFileEntry = { match: { userMessage: "test" }, diff --git a/src/__tests__/router.test.ts b/src/__tests__/router.test.ts index 50885730..8ac4396d 100644 --- a/src/__tests__/router.test.ts +++ b/src/__tests__/router.test.ts @@ -585,6 +585,127 @@ describe("matchFixture — toolCallId", () => { }); }); +// --------------------------------------------------------------------------- +// matchFixture — toolResultContains +// --------------------------------------------------------------------------- + +describe("matchFixture — toolResultContains", () => { + it("discriminates approve vs cancel legs that share a toolCallId", () => { + // The motivating case: a human-in-the-loop suspend tool resumes with the + // SAME tool_call_id for both outcomes; only the tool-result JSON differs. + const cancelled = makeFixture( + { toolCallId: "call_schedule_001", toolResultContains: '"cancelled"' }, + { content: "No problem — nothing was booked." }, + ); + const confirmed = makeFixture( + { toolCallId: "call_schedule_001", toolResultContains: '"chosen_' }, + { content: "Booked: Monday 9:00 AM confirmed." }, + ); + const base = [ + { role: "user" as const, content: "schedule a meeting" }, + { + role: "assistant" as const, + content: null, + tool_calls: [ + { + id: "call_schedule_001", + type: "function" as const, + function: { name: "schedule_meeting", arguments: "{}" }, + }, + ], + }, + ]; + const cancelReq = makeReq({ + messages: [ + ...base, + { role: "tool", content: '{"cancelled": true}', tool_call_id: "call_schedule_001" }, + ], + }); + const pickReq = makeReq({ + messages: [ + ...base, + { + role: "tool", + content: '{"chosen_time": "2026-07-20T09:00:00Z", "chosen_label": "Monday 9:00 AM"}', + tool_call_id: "call_schedule_001", + }, + ], + }); + const fixtures = [cancelled, confirmed]; + expect(matchFixture(fixtures, cancelReq)).toBe(cancelled); + expect(matchFixture(fixtures, pickReq)).toBe(confirmed); + }); + + it("matches when the last tool message contains the substring", () => { + const fixture = makeFixture({ toolResultContains: "cancelled" }); + const req = makeReq({ + messages: [ + { role: "user", content: "do thing" }, + { role: "tool", content: '{"cancelled": true}', tool_call_id: "call_x" }, + ], + }); + expect(matchFixture([fixture], req)).toBe(fixture); + }); + + it("does not match when the substring is absent", () => { + const fixture = makeFixture({ toolResultContains: "cancelled" }); + const req = makeReq({ + messages: [ + { role: "user", content: "do thing" }, + { role: "tool", content: '{"chosen_time": "9am"}', tool_call_id: "call_x" }, + ], + }); + expect(matchFixture([fixture], req)).toBeNull(); + }); + + it("does not match a request with no tool message", () => { + const fixture = makeFixture({ toolResultContains: "cancelled" }); + const req = makeReq({ messages: [{ role: "user", content: "cancelled my plans" }] }); + expect(matchFixture([fixture], req)).toBeNull(); + }); + + it("does not match when a newer user turn follows the tool message", () => { + // Same last-message rule as toolCallId: a stale tool result in history + // must not shadow matchers for the new turn. + const fixture = makeFixture({ toolResultContains: "cancelled" }); + const req = makeReq({ + messages: [ + { role: "user", content: "do thing" }, + { role: "tool", content: '{"cancelled": true}', tool_call_id: "call_x" }, + { role: "assistant", content: "Cancelled." }, + { role: "user", content: "something else" }, + ], + }); + expect(matchFixture([fixture], req)).toBeNull(); + }); + + it("matches array-of-parts tool content", () => { + const fixture = makeFixture({ toolResultContains: "cancelled" }); + const req = makeReq({ + messages: [ + { role: "user", content: "do thing" }, + { + role: "tool", + content: [{ type: "text", text: '{"cancelled": true}' }], + tool_call_id: "call_x", + }, + ], + }); + expect(matchFixture([fixture], req)).toBe(fixture); + }); + + it("does not match tool content with no extractable text", () => { + const fixture = makeFixture({ toolResultContains: "cancelled" }); + const req = makeReq({ + messages: [ + { role: "user", content: "do thing" }, + { role: "tool", content: null, tool_call_id: "call_x" }, + ], + }); + expect(matchFixture([fixture], req)).toBeNull(); + }); +}); + // --------------------------------------------------------------------------- // matchFixture — toolName // --------------------------------------------------------------------------- diff --git a/src/fixture-loader.ts b/src/fixture-loader.ts index 9d911fcb..0504b889 100644 --- a/src/fixture-loader.ts +++ b/src/fixture-loader.ts @@ -75,6 +75,7 @@ export function entryToFixture(entry: FixtureFileEntry, logger?: Logger): Fixtur systemMessage: entry.match.systemMessage, inputText: entry.match.inputText, toolCallId: entry.match.toolCallId, + toolResultContains: entry.match.toolResultContains, toolName: entry.match.toolName, model: entry.match.model, responseFormat: entry.match.responseFormat, @@ -821,6 +822,24 @@ export function validateFixtures(fixtures: Fixture[]): ValidationResult[] { message: `match.hasToolResult must be a boolean, got ${typeof f.match.hasToolResult}`, }); } + if (f.match.toolResultContains !== undefined) { + if (typeof f.match.toolResultContains !== "string") { + results.push({ + severity: "error", + fixtureIndex: i, + message: `match.toolResultContains must be a string, got ${typeof f.match.toolResultContains}`, + }); + } else if (f.match.toolResultContains.length === 0) { + // An empty substring is always contained, so the gate would silently + // act as "last message is any tool result" — reject it as an authoring + // mistake rather than let it shadow later fixtures. + results.push({ + severity: "error", + fixtureIndex: i, + message: "match.toolResultContains must be a non-empty string", + }); + } + } if (f.match.systemMessage !== undefined) { const sm = f.match.systemMessage; if (typeof sm === "string") { @@ -861,12 +880,13 @@ export function validateFixtures(fixtures: Fixture[]): ValidationResult[] { // --- Warning checks --- - // Duplicate userMessage shadowing — include turnIndex, hasToolResult, and - // sequenceIndex in the dedup key so that fixtures which share a userMessage - // but differ on those fields are NOT considered duplicates. + // Duplicate userMessage shadowing — include turnIndex, hasToolResult, + // toolResultContains, and sequenceIndex in the dedup key so that fixtures + // which share a userMessage but differ on those fields are NOT considered + // duplicates. const um = f.match.userMessage; if (typeof um === "string" && um) { - const dedupKey = `${um}|${f.match.turnIndex}|${f.match.hasToolResult}|${f.match.sequenceIndex}|${f.match.context}`; + const dedupKey = `${um}|${f.match.turnIndex}|${f.match.hasToolResult}|${f.match.toolResultContains}|${f.match.sequenceIndex}|${f.match.context}`; const prev = seenUserMessages.get(dedupKey); if (prev !== undefined) { results.push({ @@ -889,6 +909,7 @@ export function validateFixtures(fixtures: Fixture[]): ValidationResult[] { match.inputText !== undefined || match.responseFormat !== undefined || match.toolCallId !== undefined || + match.toolResultContains !== undefined || match.toolName !== undefined || match.model !== undefined || match.predicate !== undefined || diff --git a/src/router.ts b/src/router.ts index fe6ec21b..c74f465c 100644 --- a/src/router.ts +++ b/src/router.ts @@ -347,6 +347,19 @@ export function matchFixtureDiagnostic( if (!last || last.role !== "tool" || last.tool_call_id !== match.toolCallId) continue; } + // toolResultContains — substring gate on the LAST message's text content + // when that message is a tool result. Same last-message rule as toolCallId + // (a tool-result fixture answers the model's next call after a tool round), + // but discriminates on the result PAYLOAD instead of the call id — needed + // when approve/cancel resumes share the same toolCallId and differ only + // inside the tool-result JSON. + if (match.toolResultContains !== undefined) { + const last = effective.messages[effective.messages.length - 1]; + if (!last || last.role !== "tool") continue; + const text = getTextContent(last.content); + if (text === null || !text.includes(match.toolResultContains)) continue; + } + // toolName — match against any tool definition by function.name if (match.toolName !== undefined) { const tools = effective.tools ?? []; diff --git a/src/types.ts b/src/types.ts index b7a0aebe..51301447 100644 --- a/src/types.ts +++ b/src/types.ts @@ -92,6 +92,14 @@ export interface FixtureMatch { systemMessage?: string | string[] | RegExp; inputText?: string | RegExp; toolCallId?: string; + /** + * Substring matched against the text content of the LAST message when that + * message is a `tool` result (same last-message rule as `toolCallId`). + * Discriminates fixtures whose requests differ ONLY inside the tool-result + * payload — e.g. a human-in-the-loop tool where approve and cancel resume + * with the same toolCallId but different result JSON. + */ + toolResultContains?: string; toolName?: string; model?: string | RegExp; responseFormat?: string; @@ -541,6 +549,11 @@ export interface FixtureFileEntry { systemMessage?: string | string[]; inputText?: string; toolCallId?: string; + /** + * Substring matched against the last tool-result message's text content. + * Mirrors the runtime FixtureMatch.toolResultContains. + */ + toolResultContains?: string; toolName?: string; model?: string; responseFormat?: string; From 3e180bfb0474ddca37921345cbac6921c8575c4d Mon Sep 17 00:00:00 2001 From: ran Date: Wed, 15 Jul 2026 12:43:20 -0700 Subject: [PATCH 02/10] chore: 1.37.0 changelog entry (#299) --- CHANGELOG.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9f433dea..b0f95f7c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,11 @@ # @copilotkit/aimock +## [1.37.0] - 2026-07-15 + +### Added + +- New `match.toolResultContains` string gate: passes when the request's LAST message is a `role: "tool"` result whose text content contains the substring (same last-message rule as `toolCallId`, and composable with it). Discriminates fixtures whose requests differ only inside the tool-result payload — e.g. a human-in-the-loop suspend tool where approve and cancel resume with the same `tool_call_id` but different result JSON. JSON-expressible, so fixture files can finally split those legs without a programmatic `predicate`. Validated at load time (non-empty string — an empty substring would silently act as a catch-all), included in the duplicate-userMessage dedup key and the catch-all discriminator set, and routed through the `aimock-pytest` match-level kwarg keys (aimock-pytest 0.5.0) (#299) + ## [1.36.1] - 2026-07-14 ### Fixed From 985f770e01aca27ddce5ed9b37929944b3b2cf5e Mon Sep 17 00:00:00 2001 From: ran Date: Wed, 15 Jul 2026 13:41:44 -0700 Subject: [PATCH 03/10] fix: complete duplicate-userMessage dedup key with all router discriminators The dedup key in validateFixtures only carried turnIndex, hasToolResult, toolResultContains, sequenceIndex, and context, so two fixtures sharing a userMessage but legitimately disambiguated only by an omitted matcher (systemMessage, model, toolName, toolCallId, inputText, responseFormat, or endpoint) were flagged as spurious duplicates. Expand the key to every discriminator matchFixtureDiagnostic actually gates on, serialising RegExp/array matchers kind-aware so they never collide, and key predicate-bearing fixtures by identity so they are never false duplicates. The check only ever warns; it never drops or dedupes fixtures. --- CHANGELOG.md | 4 ++ src/__tests__/fixture-loader.test.ts | 84 +++++++++++++++++++++++++++ src/fixture-loader.ts | Bin 32396 -> 34707 bytes 3 files changed, 88 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index b0f95f7c..b6382778 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,10 @@ - New `match.toolResultContains` string gate: passes when the request's LAST message is a `role: "tool"` result whose text content contains the substring (same last-message rule as `toolCallId`, and composable with it). Discriminates fixtures whose requests differ only inside the tool-result payload — e.g. a human-in-the-loop suspend tool where approve and cancel resume with the same `tool_call_id` but different result JSON. JSON-expressible, so fixture files can finally split those legs without a programmatic `predicate`. Validated at load time (non-empty string — an empty substring would silently act as a catch-all), included in the duplicate-userMessage dedup key and the catch-all discriminator set, and routed through the `aimock-pytest` match-level kwarg keys (aimock-pytest 0.5.0) (#299) +### Fixed + +- The duplicate-userMessage shadow check no longer warns for fixtures that share a `userMessage` but are legitimately disambiguated by a matcher the dedup key previously omitted (`systemMessage`, `model`, `toolName`, `toolCallId`, `inputText`, `responseFormat`, `endpoint`). The dedup key now covers every discriminator the router actually matches on and serialises RegExp / array matchers kind-aware so they don't collide; a `predicate`-bearing fixture is keyed by identity so it is never a false duplicate. This only suppresses spurious warnings — the check never dropped or deduped fixtures (#299) + ## [1.36.1] - 2026-07-14 ### Fixed diff --git a/src/__tests__/fixture-loader.test.ts b/src/__tests__/fixture-loader.test.ts index 67c163d4..4bc60003 100644 --- a/src/__tests__/fixture-loader.test.ts +++ b/src/__tests__/fixture-loader.test.ts @@ -1400,6 +1400,90 @@ describe("validateFixtures", () => { expect(duplicateWarnings).toHaveLength(0); }); + it("no warning: same userMessage but different toolCallId", () => { + const fixtures = [ + makeFixture({ match: { userMessage: "hello", toolCallId: "call_a" } }), + makeFixture({ match: { userMessage: "hello", toolCallId: "call_b" } }), + ]; + const results = validateFixtures(fixtures); + const duplicateWarnings = results.filter( + (r) => r.severity === "warning" && r.message.includes("duplicate"), + ); + expect(duplicateWarnings).toHaveLength(0); + }); + + it("no warning: same userMessage but different systemMessage", () => { + const fixtures = [ + makeFixture({ match: { userMessage: "hello", systemMessage: "persona A" } }), + makeFixture({ match: { userMessage: "hello", systemMessage: "persona B" } }), + ]; + const results = validateFixtures(fixtures); + const duplicateWarnings = results.filter( + (r) => r.severity === "warning" && r.message.includes("duplicate"), + ); + expect(duplicateWarnings).toHaveLength(0); + }); + + it("no warning: same userMessage but different model", () => { + const fixtures = [ + makeFixture({ match: { userMessage: "hello", model: "gpt-4o" } }), + makeFixture({ match: { userMessage: "hello", model: "claude-opus-4" } }), + ]; + const results = validateFixtures(fixtures); + const duplicateWarnings = results.filter( + (r) => r.severity === "warning" && r.message.includes("duplicate"), + ); + expect(duplicateWarnings).toHaveLength(0); + }); + + it("no warning: same userMessage but different toolName", () => { + const fixtures = [ + makeFixture({ match: { userMessage: "hello", toolName: "search" } }), + makeFixture({ match: { userMessage: "hello", toolName: "fetch" } }), + ]; + const results = validateFixtures(fixtures); + const duplicateWarnings = results.filter( + (r) => r.severity === "warning" && r.message.includes("duplicate"), + ); + expect(duplicateWarnings).toHaveLength(0); + }); + + it("no warning: same userMessage but different responseFormat", () => { + const fixtures = [ + makeFixture({ match: { userMessage: "hello", responseFormat: "text" } }), + makeFixture({ match: { userMessage: "hello", responseFormat: "json_object" } }), + ]; + const results = validateFixtures(fixtures); + const duplicateWarnings = results.filter( + (r) => r.severity === "warning" && r.message.includes("duplicate"), + ); + expect(duplicateWarnings).toHaveLength(0); + }); + + it("no warning: same userMessage but different endpoint", () => { + const fixtures = [ + makeFixture({ match: { userMessage: "hello", endpoint: "chat" } }), + makeFixture({ match: { userMessage: "hello", endpoint: "video" } }), + ]; + const results = validateFixtures(fixtures); + const duplicateWarnings = results.filter( + (r) => r.severity === "warning" && r.message.includes("duplicate"), + ); + expect(duplicateWarnings).toHaveLength(0); + }); + + it("no warning: same userMessage but different inputText", () => { + const fixtures = [ + makeFixture({ match: { userMessage: "hello", inputText: "embed A" } }), + makeFixture({ match: { userMessage: "hello", inputText: "embed B" } }), + ]; + const results = validateFixtures(fixtures); + const duplicateWarnings = results.filter( + (r) => r.severity === "warning" && r.message.includes("duplicate"), + ); + expect(duplicateWarnings).toHaveLength(0); + }); + it("warning: same userMessage with identical turnIndex/hasToolResult/sequenceIndex", () => { const fixtures = [ makeFixture({ match: { userMessage: "hello", turnIndex: 1, hasToolResult: true } }), diff --git a/src/fixture-loader.ts b/src/fixture-loader.ts index 0504b88912324126f2e4f1599ff43c46bf84e22e..e614cef41fca2b0164fbac7fcd496939e2a39435 100644 GIT binary patch delta 2647 zcmai0U27yo6jc$F;M0ouCf8xX=`iVePl&SYCa_`>NY<4l1gFwn-BWIN)mT-XnK2uB ze}Lda_J1hsi%+bH+;j8moA+;j`{0ki?$z7dTeMAQ zTnknR!--MJY%--QR#cpZ#%dvRdO_jg`~r`sJU>{>@KDK=XF~EcA|b6J=AcrZRqOFUSQR|v~BD`>8&BBi8O9Pl(5XG)DH zKFVZ(=fQ+fP@4!47S@0KNda$i?oBQxE9l#KKmZn~9{6W+4(Jzf&2mkGL@`=OF;OCL3J=?}Ir|T^ukx}rc{r&w_0%(pxB@G69YgR%U z%Vfe;)=W5}#&V+h?)~d8d@^yE^38kqufyM2!E$pm*}HxD(PxMLRn$9z1MvLz?BqBO zgBRH{Bo)C0;oAuK3PbOHGkN{zy{~v}=c*qHh)7CeUOAT)cnnvf@tC@v8=6l!_%lVL z17X5)xIvyhI66S2pDS*x@qBh~J^TG&%F5Ul5R)ycl+!`|^x*Vavtbx{qD3hrvr79} zH{2NdAk8!%36@I*awHgaix&)IYa(Katytk|00Rul*`*btAg`qX{TtqlYJkit`vi0z zwP04M;vpa&rz4PA6{p-(g?$)S43)We$JoXZl`7@M2t}Kjl7=5DjZ8&5?#S-LLv?YNs=%?vsGIZTlSXoDUKePlq z5Lgt)I}bWu!yDhNWxQ246JASf+~gaPryV(t6g(GJpgdkIciQFbne+F{SH$BycD7v~ zoyw&m%dp-j$KO43tLbtGy(P@JR5cn%&KWD8u>yMAa}8_j>DS^-W}2sd^(Rg|8Fd@& zdY3TiwC<8a?#daMoKl7mXt)63UDG`$g%jQuQS$l<$z^CG%!6c;nR;imb=SoR8UlE& zmwp!1rQx!$8iK{s77=AYW|U0PEG0b;yJigm8FWlo5OXdO5%n)3r~SxFa%er$K?13o z_er7(1PX71KSQO?vP}-6=Gd^pb7Bi&HXHcVoiH>GTi$4(sR7tuA(u#VcgFpJ^V`QY zG})h%w$$dT(2#Zagf@;zxFy{!!)0)l2W|7$gZY@6Lkg z8|sM%^%OU_a^24^ov-}mUf;PcWxO(%$f=B9x)$Os_G=R5JzC@c3D9ZS9pf_YH!cVq zAUV_n40F`ksk!#Ix6KTcrj?bD#y(Q~b_U?XK z)4}!67e8#o0Z(Sb;Z;QYU*oc`Uv+R{FrZyJ^Zx|GD2g_{9eIa?7lY{atL48R{s(N# BVDSI| delta 318 zcmYk1u}T9$5Qb4xSqhiJYJ@bxkvu^vD-jhXolVx=|J=pPZrs_$lNbWlg2k0Sgn=|x zc^4nS&PH(WAi-${KK}VXrheOf{oPrv4_2Sq5f;YEGAdAr-kw^8Dkm*?K?NmxaUhX^at9+R;YG)O_kDc$3V*d-tW6{boG@TTrj8>xw}wD2H7 zk>St5Ebc(JF_1Dv0Z*=}bBuJ3QZY5*i^~y7XW1xts7+t;l}3ty`eSi(-1^Uwen{iq uEKcUny;;$ph#+-;`)LE~rSI-F`^z78TJSy1vj6d<3aG+-yn286>HGnsM09ol From 87935ea3b7ddd1e9ac8dff558bf837aa4c455576 Mon Sep 17 00:00:00 2001 From: ran Date: Wed, 15 Jul 2026 15:18:44 -0700 Subject: [PATCH 04/10] docs: document toolResultContains and correct match/dedup/toolCallId rules Consolidates the match-field, dedup-caveat, and toolCallId-rule doc corrections for the new gate into one change across docs/fixtures, docs/multi-turn, and the write-fixtures skill. --- docs/fixtures/index.html | 46 +++++++++++++++++++--------- docs/multi-turn/index.html | 55 +++++++++++++++++++++++----------- skills/write-fixtures/SKILL.md | 2 +- 3 files changed, 71 insertions(+), 32 deletions(-) diff --git a/docs/fixtures/index.html b/docs/fixtures/index.html index 7852e0e4..3cd6b872 100644 --- a/docs/fixtures/index.html +++ b/docs/fixtures/index.html @@ -97,6 +97,16 @@

    Match Fields

    requestTransform is set) or regex (pattern match) + + systemMessage + string | RegExp | string[] + + Match on the joined text of the request’s system messages — string + (case-sensitive substring, or exact when requestTransform is set), + regex (pattern match), or an array of strings (all substrings must be present). Gate + a fixture on host-supplied context + + inputText string | RegExp @@ -210,12 +220,16 @@

    1. userMessage matches only the LAST user message

    tool-round idiom.

    -

    2. toolCallId matches the LAST tool message

    +

    2. toolCallId matches the overall last message

    - toolCallId is compared against the tool_call_id of the - last role: "tool" message in the request — regardless of - whether that’s the overall last message. If no tool message is present in the - history, toolCallId never matches. See + toolCallId matches only when the request’s + overall last message is a role: "tool" result whose + tool_call_id equals the given value. This mirrors the API contract: a tool + result is the last message only while the model is answering that tool round. If the last + message is anything else — a newer user turn, or no tool result at all + — toolCallId never matches, so a stale tool round can’t shadow + userMessage matchers. This is the same overall-last-message rule as + toolResultContains. See Multi-Turn Conversations for the tool-round idiom.

    @@ -262,15 +276,19 @@

    5. Validation warnings surface shadowing at load time

    duplicate userMessage 'hello' — shadows fixture 0, where 'hello' is the duplicated message and 0 is the zero-based index of the earlier fixture being shadowed. This is advisory, not a hard error: the - check now factors in turnIndex, hasToolResult, - context, and sequenceIndex when deciding whether two fixtures - truly collide, but it does not consider toolCallId, - model, or predicate, so the warning may still fire when those - discriminators are present. Treat it as advisory: if a runtime differentiator is in - place, the fixtures won't actually shadow each other at match time. Only fixtures with - no differentiator at all will truly shadow on match — that's the case where the - second is never reached because the first wins. Safe to ignore in the former case; - investigate in the latter. + check factors in every match discriminator the router gates on when deciding + whether two fixtures truly collide — userMessage, + systemMessage, inputText, toolCallId, + toolResultContains, toolName, model, + responseFormat, endpoint, context, + sequenceIndex, turnIndex, and + hasToolResult — so two fixtures that differ on any one of them do not + trigger the warning. A predicate is a function that cannot be compared for + equivalence, so any fixture carrying one is keyed by its own identity: it is treated as + unconditionally distinct and never reported as a duplicate (and two fixtures with + distinct predicates are likewise treated as distinct). Only fixtures with no + differentiator at all will truly shadow on match — that's the case where the + second is never reached because the first wins. That is the case worth investigating.
  • Catch-all not last — a fixture with an empty match diff --git a/docs/multi-turn/index.html b/docs/multi-turn/index.html index 3982e73c..0d98c741 100644 --- a/docs/multi-turn/index.html +++ b/docs/multi-turn/index.html @@ -80,7 +80,8 @@

    How matching works across turns

    human-in-the-loop tool where approve resumes with {"chosen_time": …} and cancel resumes with {"cancelled": true}. Gate the cancel fixture on - "toolResultContains": "cancelled" and order it before the approve leg. + "toolResultContains": "\"cancelled\"" (match the quoted JSON key so it + can’t match the word inside prose) and order it before the approve leg.

    • @@ -110,8 +111,10 @@

    How matching works across turns

    Substring by default, exact when transformed. userMessage is a substring match by default ("hello" matches "say hello world"). When you register a requestTransform, - matching flips to exact string equality — but only for - userMessage and inputText; other fields like + matching flips to exact string equality — but only for the + string forms of userMessage, systemMessage, and + inputText. Regex and array-of-substrings forms are unaffected, and + toolResultContains stays a substring match. Fields like toolName and toolCallId are always exact. This trips people up — see Gotchas below.

    @@ -302,10 +305,10 @@

    Turn 2 — client runs the tool, sends the result

    - Choosing between sequenceIndex, toolCallId, turnIndex, hasToolResult, context, and - predicate + Choosing between sequenceIndex, toolCallId, toolResultContains, turnIndex, hasToolResult, + context, and predicate

    -

    Five mechanisms handle different shapes of “the same prompt twice”:

    +

    Seven mechanisms handle different shapes of “the same prompt twice”:

    @@ -328,8 +331,20 @@

    + + + + + @@ -430,19 +445,25 @@

    Gotchas

    Substring vs. exact matching. Default matching is substring. Adding a requestTransform (e.g. to strip timestamps or request ids) flips matching to exact string equality — fixtures that previously matched as substrings will - silently stop matching. Only userMessage and inputText flip; - fields like toolName and toolCallId are always exact. Pin - exact strings in your fixtures when you use a transform. + silently stop matching. Only the string forms of userMessage, + systemMessage, and inputText flip; regex and + array-of-substrings forms are unaffected, and toolResultContains remains a + substring match even under a transform. Fields like toolName and + toolCallId are always exact. Pin exact strings in your fixtures when you + use a transform.
  • Duplicate userMessage warnings. validateFixtures warns when two fixtures share the same - userMessage with identical turnIndex, - hasToolResult, and sequenceIndex values. Fixtures that differ - on any of these discriminators do not trigger the warning. Other fields like - toolCallId, model, and predicate are not factored - in, so the warning may still fire when those discriminators are present. Treat it as - advisory. + userMessage with identical values for every match discriminator + the router gates on — userMessage, systemMessage, + inputText, toolCallId, toolResultContains, + toolName, model, responseFormat, + endpoint, context, sequenceIndex, + turnIndex, and hasToolResult. Fixtures that differ on any one + of these discriminators do not trigger the warning. A predicate cannot be + compared for equivalence, so any fixture carrying one is keyed by its own identity and + is never reported as a duplicate. Treat it as advisory.
  • First-wins ordering. Fixtures are evaluated in registration order (and, diff --git a/skills/write-fixtures/SKILL.md b/skills/write-fixtures/SKILL.md index 72906c51..4aa0a46a 100644 --- a/skills/write-fixtures/SKILL.md +++ b/skills/write-fixtures/SKILL.md @@ -54,7 +54,7 @@ Multi-part content (e.g., `[{type: "text", text: "hello"}]`) is automatically ex | `toolCallId` | Yes | Matching specific tool result IDs in the conversation history | | `toolResultContains` | Yes | Same tool call id, different outcomes — match on the tool-result payload (approve vs cancel legs) | -**Prefer stateless approaches** (`turnIndex`, `hasToolResult`) for shared aimock instances (deployed via Docker, used by multiple test runners). Use `sequenceIndex` only in isolated single-client unit tests where the counter won't be corrupted by concurrent requests. +**Prefer stateless approaches** (`turnIndex`, `hasToolResult`, `toolResultContains`) for shared aimock instances (deployed via Docker, used by multiple test runners). Use `sequenceIndex` only in isolated single-client unit tests where the counter won't be corrupted by concurrent requests. ### Multi-turn fixture examples From 867835667035cf83e5ef42fececbb56bd15af588 Mon Sep 17 00:00:00 2001 From: ran Date: Wed, 15 Jul 2026 13:39:39 -0700 Subject: [PATCH 05/10] chore: drop redundant requests dep from aimock-pytest test extra requests is already a main dependency so it is always installed; listing it again in the test optional-dependency group is redundant. --- packages/aimock-pytest/pyproject.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/aimock-pytest/pyproject.toml b/packages/aimock-pytest/pyproject.toml index 39666a5c..1cbd98c0 100644 --- a/packages/aimock-pytest/pyproject.toml +++ b/packages/aimock-pytest/pyproject.toml @@ -12,7 +12,7 @@ license = "MIT" dependencies = ["requests>=2.28"] [project.optional-dependencies] -test = ["pytest>=7.0", "requests>=2.28"] +test = ["pytest>=7.0"] [project.entry-points."pytest11"] aimock = "aimock_pytest.plugin" From 7c428d5ec11ea8f7c8f3efcba168aaa8739d6fcd Mon Sep 17 00:00:00 2001 From: ran Date: Wed, 15 Jul 2026 14:01:45 -0700 Subject: [PATCH 06/10] fix: bump aimock-pytest default server pin to 1.37.0 The auto-download path (AIMOCK_CLI_PATH unset) fetched 1.35.1, which predates the server toolResultContains route. Since 0.5.0 forwards that match kwarg, default pip users got a server that silently ignored it. Pin now tracks the 1.37.0 release that adds the route. --- CHANGELOG.md | 1 + packages/aimock-pytest/src/aimock_pytest/_version.py | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b6382778..10f3ef74 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ ### Fixed - The duplicate-userMessage shadow check no longer warns for fixtures that share a `userMessage` but are legitimately disambiguated by a matcher the dedup key previously omitted (`systemMessage`, `model`, `toolName`, `toolCallId`, `inputText`, `responseFormat`, `endpoint`). The dedup key now covers every discriminator the router actually matches on and serialises RegExp / array matchers kind-aware so they don't collide; a `predicate`-bearing fixture is keyed by identity so it is never a false duplicate. This only suppresses spurious warnings — the check never dropped or deduped fixtures (#299) +- Bumped the `aimock-pytest` default server pin (`_version.py`) to 1.37.0 so the auto-download path (used when `AIMOCK_CLI_PATH` is unset) runs a server that supports the new `toolResultContains` match kwarg the client forwards (#299) ## [1.36.1] - 2026-07-14 diff --git a/packages/aimock-pytest/src/aimock_pytest/_version.py b/packages/aimock-pytest/src/aimock_pytest/_version.py index e1f62ee5..9c863968 100644 --- a/packages/aimock-pytest/src/aimock_pytest/_version.py +++ b/packages/aimock-pytest/src/aimock_pytest/_version.py @@ -8,4 +8,4 @@ control routes ship in the next release). Keep it tracking npm releases. """ -AIMOCK_VERSION = "1.35.1" +AIMOCK_VERSION = "1.37.0" From eace7b050a83361a7f028a2a50574b121f402415 Mon Sep 17 00:00:00 2001 From: Jordan Ritter Date: Wed, 15 Jul 2026 15:47:08 -0700 Subject: [PATCH 07/10] ci: add format:check and test:exports to release gate The release script published without verifying formatting or the package export map, so unformatted code or a broken export map could reach npm (recurring failure, see CHANGELOG #263). Add format:check before build (cheap, fail fast) and test:exports after build (needs the built dist), with npm publish last. --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index 87b8165d..b2fb7a41 100644 --- a/package.json +++ b/package.json @@ -169,7 +169,7 @@ "test:exports": "publint && attw --pack .", "lint": "eslint .", "format:check": "prettier --check .", - "release": "pnpm build && pnpm test && pnpm lint && npm publish", + "release": "pnpm format:check && pnpm build && pnpm test && pnpm lint && pnpm test:exports && npm publish", "prepare": "husky || true" }, "lint-staged": { From 331cbacebaee97686b8cbc58733131473760a622 Mon Sep 17 00:00:00 2001 From: Jordan Ritter Date: Wed, 15 Jul 2026 15:48:31 -0700 Subject: [PATCH 08/10] docs: fix E2E fixture lifecycle to use resetMatchCounts The E2E setup examples loaded fixtures once then called full reset() in afterEach, which wipes the in-memory fixture pool and 404s every test after the first. Switch between-test cleanup to resetMatchCounts() (suite.llm.resetMatchCounts() for MockSuite) and correct the prose so it leads with resetMatchCounts() and warns reset() also clears fixtures. --- skills/write-fixtures/SKILL.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/skills/write-fixtures/SKILL.md b/skills/write-fixtures/SKILL.md index 4aa0a46a..c9c7e05a 100644 --- a/skills/write-fixtures/SKILL.md +++ b/skills/write-fixtures/SKILL.md @@ -262,7 +262,7 @@ mock.on( mock.on({ userMessage: "status", sequenceIndex: 1 }, { content: "All systems operational." }); ``` -Match counts are tracked per fixture group and reset with `reset()` or `resetMatchCounts()`. +Match counts are tracked per fixture group. Use `resetMatchCounts()` between tests to reset counts while keeping loaded fixtures. `reset()` also clears the fixture pool, so avoid it between tests that share a loaded fixture set. ### Streaming physics (realistic timing) @@ -507,7 +507,7 @@ These fields map correctly across all provider formats — for example, `finishR 10. **Embeddings auto-generate if no fixture matches** — deterministic vectors are generated from the input text hash. You don't need a catch-all for embedding requests. -11. **Sequential response counts are tracked per fixture** — counts reset with `reset()` or `resetMatchCounts()`. The count increments after each match of that fixture group (all fixtures sharing the same non-`sequenceIndex` match fields). +11. **Sequential response counts are tracked per fixture** — use `resetMatchCounts()` between tests to reset counts while keeping loaded fixtures; `reset()` also clears the fixture pool, so don't use it between tests that share a loaded fixture set. The count increments after each match of that fixture group (all fixtures sharing the same non-`sequenceIndex` match fields). 12. **Bedrock uses Anthropic Messages format internally** — the adapter normalizes Bedrock requests to `ChatCompletionRequest`, so the same fixtures work. Bedrock supports both non-streaming (`/invoke`, `/converse`) and streaming (`/invoke-with-response-stream`, `/converse-stream`) endpoints. @@ -556,7 +556,7 @@ await suite.start(); // suite.llm — the LLMock instance // suite.url — base URL -afterEach(() => suite.reset()); // resets everything +afterEach(() => suite.llm.resetMatchCounts()); // reset sequence counts, keep fixtures afterAll(() => suite.stop()); ``` @@ -691,8 +691,8 @@ mock.loadFixtureDir("./fixtures"); await mock.start(); process.env.OPENAI_BASE_URL = `${mock.url}/v1`; -// Per-test cleanup -afterEach(() => mock.reset()); // clears fixtures AND journal +// Per-test cleanup — reset sequence match counts, keep the loaded fixtures +afterEach(() => mock.resetMatchCounts()); // Teardown afterAll(async () => await mock.stop()); @@ -743,6 +743,8 @@ const mock = await LLMock.create({ port: 0 }); // creates + starts in one call | `url` / `baseUrl` | Server URL (throws if not started) | | `port` | Server port number | +Between tests that share a loaded fixture set, use `resetMatchCounts()` (not `reset()`, which also clears fixtures). For a `MockSuite`, call `suite.llm.resetMatchCounts()` — the suite itself has no `resetMatchCounts()`. + Sequential responses use `on()` with `sequenceIndex` in the match — there is no dedicated convenience method. ## Record-and-Replay (VCR Mode) From 0d461bda953e00dae0a69dc011678d29c7472f54 Mon Sep 17 00:00:00 2001 From: Jordan Ritter Date: Wed, 15 Jul 2026 15:49:25 -0700 Subject: [PATCH 09/10] fix: guard recordedTimings.interChunkDelaysMs against non-array values Untrusted fixture JSON may set interChunkDelaysMs to null/missing/non-array. The sanitizer called .filter with no Array.isArray guard, throwing a TypeError inside entryToFixture that aborted the entire fixture file's load and silently dropped every fixture in it. Coerce non-array values to [] like sibling guards. --- CHANGELOG.md | 1 + src/__tests__/fixture-loader.test.ts | 26 ++++++++++++++++++++++++++ src/fixture-loader.ts | Bin 34707 -> 34763 bytes 3 files changed, 27 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 10f3ef74..897bfb70 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ - The duplicate-userMessage shadow check no longer warns for fixtures that share a `userMessage` but are legitimately disambiguated by a matcher the dedup key previously omitted (`systemMessage`, `model`, `toolName`, `toolCallId`, `inputText`, `responseFormat`, `endpoint`). The dedup key now covers every discriminator the router actually matches on and serialises RegExp / array matchers kind-aware so they don't collide; a `predicate`-bearing fixture is keyed by identity so it is never a false duplicate. This only suppresses spurious warnings — the check never dropped or deduped fixtures (#299) - Bumped the `aimock-pytest` default server pin (`_version.py`) to 1.37.0 so the auto-download path (used when `AIMOCK_CLI_PATH` is unset) runs a server that supports the new `toolResultContains` match kwarg the client forwards (#299) +- Guarded `recordedTimings.interChunkDelaysMs` against non-array values: a fixture whose `interChunkDelaysMs` is malformed (null, missing, or not an array) no longer throws during load and silently drops every fixture in that file — the value is coerced to an empty array (#299) ## [1.36.1] - 2026-07-14 diff --git a/src/__tests__/fixture-loader.test.ts b/src/__tests__/fixture-loader.test.ts index 4bc60003..1617eda2 100644 --- a/src/__tests__/fixture-loader.test.ts +++ b/src/__tests__/fixture-loader.test.ts @@ -316,6 +316,32 @@ describe("loadFixtureFile", () => { expect(fixtures[0].disconnectAfterMs).toBeUndefined(); }); + it("loads sibling fixtures when one entry has a non-array interChunkDelaysMs", () => { + // Untrusted fixture JSON: interChunkDelaysMs is typed number[] but may be + // null/missing/non-array. Without an Array.isArray guard, .filter throws a + // TypeError inside entryToFixture, aborting the entire file's load and + // silently dropping every fixture in it. + const filePath = writeJson(tmpDir, "bad-timings.json", { + fixtures: [ + { + match: { userMessage: "malformed" }, + response: { content: "bad" }, + recordedTimings: { ttftMs: 0, interChunkDelaysMs: null, totalDurationMs: 0 }, + }, + { + match: { userMessage: "valid" }, + response: { content: "good" }, + }, + ], + }); + + const fixtures = loadFixtureFile(filePath); + expect(fixtures).toHaveLength(2); + expect(fixtures[1].match.userMessage).toBe("valid"); + // The malformed entry is sanitized: non-array interChunkDelaysMs -> []. + expect(fixtures[0].recordedTimings?.interChunkDelaysMs).toEqual([]); + }); + it("warns and returns empty array for invalid JSON", () => { const filePath = join(tmpDir, "bad.json"); writeFileSync(filePath, "{ not valid json", "utf-8"); diff --git a/src/fixture-loader.ts b/src/fixture-loader.ts index e614cef41fca2b0164fbac7fcd496939e2a39435..4fe26366f7e54d2c737ec9455fa69208f15338e1 100644 GIT binary patch delta 63 zcmbQ-&vd$e|1*{aJV>ed` HN|pftO~w#< delta 26 icmX@z&osH8X@jNUWJN*I$<=~7ldlQLZ@wt_s}ul(zY076 From 5a395a5a9d7e1cfdde1a4396b21f02364fc7f73d Mon Sep 17 00:00:00 2001 From: ran Date: Wed, 15 Jul 2026 17:36:00 -0700 Subject: [PATCH 10/10] docs: include systemMessage in requestTransform exact-match field list The fixtures and record-replay pages stated that only string userMessage and inputText flip to strict equality under a requestTransform, omitting systemMessage. The router flips all three string forms (userMessage line 284, systemMessage line 330, inputText line 377). Align both pages with the code and the multi-turn page's already-correct field list. --- docs/fixtures/index.html | 8 ++++---- docs/record-replay/index.html | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/fixtures/index.html b/docs/fixtures/index.html index 3cd6b872..acbfc9e2 100644 --- a/docs/fixtures/index.html +++ b/docs/fixtures/index.html @@ -258,10 +258,10 @@

    4. Substring by default, exact when a requestTransform is set If the router is configured with a requestTransform (typically used to strip dynamic data like timestamps or UUIDs from the request before matching), string - userMessage and inputText flip to strict equality - (===). The rationale: transforms normalize requests to a canonical form, and - once normalized, the sensible comparison is exact — substring matching on a - normalized string is more likely to hide bugs than catch flexible input. + userMessage, systemMessage, and inputText flip to + strict equality (===). The rationale: transforms normalize requests to a + canonical form, and once normalized, the sensible comparison is exact — substring + matching on a normalized string is more likely to hide bugs than catch flexible input.

    5. Validation warnings surface shadowing at load time

    diff --git a/docs/record-replay/index.html b/docs/record-replay/index.html index 6d49dfd4..d5974b30 100644 --- a/docs/record-replay/index.html +++ b/docs/record-replay/index.html @@ -1007,8 +1007,8 @@

    Request Transform

    - When requestTransform is set, string matching for - userMessage and inputText switches from substring + When requestTransform is set, string matching for userMessage, + systemMessage, and inputText switches from substring (includes) to exact equality (===). This prevents shortened keys from accidentally matching unrelated prompts. Without a transform, the existing includes behavior is preserved for backward compatibility.

    • Different behavior before vs. after tool execution (tool-call round trip) toolCallId - Matches the tool_call_id of the last role: "tool" - message. Turn 1 has no tool message; turn 2 does. + Matches when the request’s overall last message is a + role: "tool" result with that tool_call_id. Turn 1 has no + tool result last; turn 2 does. +
    + Two tool-result outcomes that share a tool_call_id and differ only + inside the result payload (e.g. approve vs. cancel) + toolResultContains + Substring match on the last role: "tool" result’s text. Use when + toolCallId alone can’t tell the outcomes apart. Stateless.