diff --git a/CHANGELOG.md b/CHANGELOG.md index 9f433dea..897bfb70 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,17 @@ # @copilotkit/aimock +## [1.37.0] - 2026-07-15 + +### Added + +- New `match.toolResultContains` string gate: passes when the request's LAST message is a `role: "tool"` result whose text content contains the substring (same last-message rule as `toolCallId`, and composable with it). Discriminates fixtures whose requests differ only inside the tool-result payload — e.g. a human-in-the-loop suspend tool where approve and cancel resume with the same `tool_call_id` but different result JSON. JSON-expressible, so fixture files can finally split those legs without a programmatic `predicate`. Validated at load time (non-empty string — an empty substring would silently act as a catch-all), included in the duplicate-userMessage dedup key and the catch-all discriminator set, and routed through the `aimock-pytest` match-level kwarg keys (aimock-pytest 0.5.0) (#299) + +### Fixed + +- The duplicate-userMessage shadow check no longer warns for fixtures that share a `userMessage` but are legitimately disambiguated by a matcher the dedup key previously omitted (`systemMessage`, `model`, `toolName`, `toolCallId`, `inputText`, `responseFormat`, `endpoint`). The dedup key now covers every discriminator the router actually matches on and serialises RegExp / array matchers kind-aware so they don't collide; a `predicate`-bearing fixture is keyed by identity so it is never a false duplicate. This only suppresses spurious warnings — the check never dropped or deduped fixtures (#299) +- Bumped the `aimock-pytest` default server pin (`_version.py`) to 1.37.0 so the auto-download path (used when `AIMOCK_CLI_PATH` is unset) runs a server that supports the new `toolResultContains` match kwarg the client forwards (#299) +- Guarded `recordedTimings.interChunkDelaysMs` against non-array values: a fixture whose `interChunkDelaysMs` is malformed (null, missing, or not an array) no longer throws during load and silently drops every fixture in that file — the value is coerced to an empty array (#299) + ## [1.36.1] - 2026-07-14 ### Fixed diff --git a/README.md b/README.md index 687ca280..4203bd44 100644 --- a/README.md +++ b/README.md @@ -50,7 +50,7 @@ Run them all on one port with `npx @copilotkit/aimock --config aimock.json`, or - **[Record & Replay](https://aimock.copilotkit.dev/record-replay)** — Proxy real APIs, save as fixtures, replay deterministically forever - **Timing-aware recording and replay** — Recorded fixtures capture per-frame arrival timestamps; replay uses recorded timings for approximate timing reproduction based on recorded TTFT and inter-frame cadence (replay chunk count may differ from recording — TTFT and average pace are preserved, not per-token fidelity) with configurable `--replay-speed` multiplier -- **[Multi-turn Conversations](https://aimock.copilotkit.dev/multi-turn)** — Record and replay multi-turn traces with tool rounds; match distinct turns via `turnIndex`, `hasToolResult`, `toolCallId`, `sequenceIndex`, `systemMessage` (gate on host-supplied agent context), or custom predicates +- **[Multi-turn Conversations](https://aimock.copilotkit.dev/multi-turn)** — Record and replay multi-turn traces with tool rounds; match distinct turns via `turnIndex`, `hasToolResult`, `toolCallId`, `toolResultContains` (gate on the tool-result payload), `sequenceIndex`, `systemMessage` (gate on host-supplied agent context), or custom predicates - **[12 providers across 14 API surfaces](https://aimock.copilotkit.dev/docs)** — OpenAI Chat, OpenAI Responses, OpenAI Realtime (GA + Beta shim), Claude, Gemini (REST + embedContent), Gemini Live, Gemini Interactions, Azure, Bedrock, Vertex AI, Ollama (chat + embeddings), Cohere (chat + embed), ElevenLabs TTS — full streaming support - **Multimedia APIs** — [image generation](https://aimock.copilotkit.dev/images) (DALL-E, Imagen), [image editing](https://aimock.copilotkit.dev/images) (/v1/images/edits), [text-to-speech](https://aimock.copilotkit.dev/speech) (OpenAI + ElevenLabs), [audio transcription](https://aimock.copilotkit.dev/transcription), [audio translation](https://aimock.copilotkit.dev/transcription) (/v1/audio/translations), [video generation](https://aimock.copilotkit.dev/video), [OpenRouter video generation](https://aimock.copilotkit.dev/openrouter-video) (/api/v1/videos with async job lifecycle), [Google Veo video generation](https://aimock.copilotkit.dev/veo-video) (:predictLongRunning + /v1beta/operations async lifecycle), [Grok Imagine video generation](https://aimock.copilotkit.dev/grok-video) (/v1/videos/generations with async job lifecycle), [fal.ai](https://aimock.copilotkit.dev/fal-ai) (image / video / audio with queue lifecycle) - **[MCP](https://aimock.copilotkit.dev/mcp-mock) / [A2A](https://aimock.copilotkit.dev/a2a-mock) / [AG-UI](https://aimock.copilotkit.dev/agui-mock) / [Vector](https://aimock.copilotkit.dev/vector-mock)** — Mock every protocol your AI agents use diff --git a/docs/fixtures/index.html b/docs/fixtures/index.html index 67a0b2d9..acbfc9e2 100644 --- a/docs/fixtures/index.html +++ b/docs/fixtures/index.html @@ -97,6 +97,16 @@

Match Fields

requestTransform is set) or regex (pattern match) + + systemMessage + string | RegExp | string[] + + Match on the joined text of the request’s system messages — string + (case-sensitive substring, or exact when requestTransform is set), + regex (pattern match), or an array of strings (all substrings must be present). Gate + a fixture on host-supplied context + + inputText string | RegExp @@ -110,6 +120,16 @@

Match Fields

The onToolResult(id, response) helper is sugar over this field + + toolResultContains + string + + Substring match on the text content of the last message when it is a + role: "tool" result. Discriminates fixtures whose requests differ only + inside the tool-result payload (e.g. approve vs cancel resuming with the same + tool_call_id). See Multi-Turn Conversations + + toolName string @@ -200,12 +220,16 @@

1. userMessage matches only the LAST user message

tool-round idiom.

-

2. toolCallId matches the LAST tool message

+

2. toolCallId matches the overall last message

- toolCallId is compared against the tool_call_id of the - last role: "tool" message in the request — regardless of - whether that’s the overall last message. If no tool message is present in the - history, toolCallId never matches. See + toolCallId matches only when the request’s + overall last message is a role: "tool" result whose + tool_call_id equals the given value. This mirrors the API contract: a tool + result is the last message only while the model is answering that tool round. If the last + message is anything else — a newer user turn, or no tool result at all + — toolCallId never matches, so a stale tool round can’t shadow + userMessage matchers. This is the same overall-last-message rule as + toolResultContains. See Multi-Turn Conversations for the tool-round idiom.

@@ -234,10 +258,10 @@

4. Substring by default, exact when a requestTransform is set If the router is configured with a requestTransform (typically used to strip dynamic data like timestamps or UUIDs from the request before matching), string - userMessage and inputText flip to strict equality - (===). The rationale: transforms normalize requests to a canonical form, and - once normalized, the sensible comparison is exact — substring matching on a - normalized string is more likely to hide bugs than catch flexible input. + userMessage, systemMessage, and inputText flip to + strict equality (===). The rationale: transforms normalize requests to a + canonical form, and once normalized, the sensible comparison is exact — substring + matching on a normalized string is more likely to hide bugs than catch flexible input.

5. Validation warnings surface shadowing at load time

@@ -252,15 +276,19 @@

5. Validation warnings surface shadowing at load time

duplicate userMessage 'hello' — shadows fixture 0, where 'hello' is the duplicated message and 0 is the zero-based index of the earlier fixture being shadowed. This is advisory, not a hard error: the - check now factors in turnIndex, hasToolResult, - context, and sequenceIndex when deciding whether two fixtures - truly collide, but it does not consider toolCallId, - model, or predicate, so the warning may still fire when those - discriminators are present. Treat it as advisory: if a runtime differentiator is in - place, the fixtures won't actually shadow each other at match time. Only fixtures with - no differentiator at all will truly shadow on match — that's the case where the - second is never reached because the first wins. Safe to ignore in the former case; - investigate in the latter. + check factors in every match discriminator the router gates on when deciding + whether two fixtures truly collide — userMessage, + systemMessage, inputText, toolCallId, + toolResultContains, toolName, model, + responseFormat, endpoint, context, + sequenceIndex, turnIndex, and + hasToolResult — so two fixtures that differ on any one of them do not + trigger the warning. A predicate is a function that cannot be compared for + equivalence, so any fixture carrying one is keyed by its own identity: it is treated as + unconditionally distinct and never reported as a duplicate (and two fixtures with + distinct predicates are likewise treated as distinct). Only fixtures with no + differentiator at all will truly shadow on match — that's the case where the + second is never reached because the first wins. That is the case worth investigating.
  • Catch-all not last — a fixture with an empty match diff --git a/docs/multi-turn/index.html b/docs/multi-turn/index.html index 3b56a788..0d98c741 100644 --- a/docs/multi-turn/index.html +++ b/docs/multi-turn/index.html @@ -72,6 +72,17 @@

    How matching works across turns

    distinguish the turn that requests a tool from the turn that follows up on a tool result.
    • +
  • + toolResultContains (v1.37.0) is a substring match against the + text content of the last tool message (same last-message rule as + toolCallId). Use it when two resume paths share the same + tool_call_id and differ only inside the tool-result payload — e.g. a + human-in-the-loop tool where approve resumes with + {"chosen_time": …} and cancel resumes with + {"cancelled": true}. Gate the cancel fixture on + "toolResultContains": "\"cancelled\"" (match the quoted JSON key so it + can’t match the word inside prose) and order it before the approve leg. +

    • A request carrying a 20-message history still only matches on its last user message (and, @@ -100,8 +111,10 @@

    How matching works across turns

    Substring by default, exact when transformed. userMessage is a substring match by default ("hello" matches "say hello world"). When you register a requestTransform, - matching flips to exact string equality — but only for - userMessage and inputText; other fields like + matching flips to exact string equality — but only for the + string forms of userMessage, systemMessage, and + inputText. Regex and array-of-substrings forms are unaffected, and + toolResultContains stays a substring match. Fields like toolName and toolCallId are always exact. This trips people up — see Gotchas below.

    @@ -292,10 +305,10 @@

    Turn 2 — client runs the tool, sends the result

    - Choosing between sequenceIndex, toolCallId, turnIndex, hasToolResult, context, and - predicate + Choosing between sequenceIndex, toolCallId, toolResultContains, turnIndex, hasToolResult, + context, and predicate

    -

    Five mechanisms handle different shapes of “the same prompt twice”:

    +

    Seven mechanisms handle different shapes of “the same prompt twice”:

    @@ -318,8 +331,20 @@

    + + + + + @@ -420,19 +445,25 @@

    Gotchas

    Substring vs. exact matching. Default matching is substring. Adding a requestTransform (e.g. to strip timestamps or request ids) flips matching to exact string equality — fixtures that previously matched as substrings will - silently stop matching. Only userMessage and inputText flip; - fields like toolName and toolCallId are always exact. Pin - exact strings in your fixtures when you use a transform. + silently stop matching. Only the string forms of userMessage, + systemMessage, and inputText flip; regex and + array-of-substrings forms are unaffected, and toolResultContains remains a + substring match even under a transform. Fields like toolName and + toolCallId are always exact. Pin exact strings in your fixtures when you + use a transform.
  • Duplicate userMessage warnings. validateFixtures warns when two fixtures share the same - userMessage with identical turnIndex, - hasToolResult, and sequenceIndex values. Fixtures that differ - on any of these discriminators do not trigger the warning. Other fields like - toolCallId, model, and predicate are not factored - in, so the warning may still fire when those discriminators are present. Treat it as - advisory. + userMessage with identical values for every match discriminator + the router gates on — userMessage, systemMessage, + inputText, toolCallId, toolResultContains, + toolName, model, responseFormat, + endpoint, context, sequenceIndex, + turnIndex, and hasToolResult. Fixtures that differ on any one + of these discriminators do not trigger the warning. A predicate cannot be + compared for equivalence, so any fixture carrying one is keyed by its own identity and + is never reported as a duplicate. Treat it as advisory.
  • First-wins ordering. Fixtures are evaluated in registration order (and, diff --git a/docs/record-replay/index.html b/docs/record-replay/index.html index 6d49dfd4..d5974b30 100644 --- a/docs/record-replay/index.html +++ b/docs/record-replay/index.html @@ -1007,8 +1007,8 @@

    Request Transform

    - When requestTransform is set, string matching for - userMessage and inputText switches from substring + When requestTransform is set, string matching for userMessage, + systemMessage, and inputText switches from substring (includes) to exact equality (===). This prevents shortened keys from accidentally matching unrelated prompts. Without a transform, the existing includes behavior is preserved for backward compatibility. diff --git a/package.json b/package.json index 8fd650c2..b2fb7a41 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@copilotkit/aimock", - "version": "1.36.1", + "version": "1.37.0", "description": "Mock infrastructure for AI application testing — LLM APIs, image generation, image editing, text-to-speech, transcription, audio translation, audio generation, video generation, embeddings, MCP tools, A2A agents, AG-UI event streams, vector databases, search, rerank, and moderation. One package, one port, zero dependencies.", "license": "MIT", "keywords": [ @@ -169,7 +169,7 @@ "test:exports": "publint && attw --pack .", "lint": "eslint .", "format:check": "prettier --check .", - "release": "pnpm build && pnpm test && pnpm lint && npm publish", + "release": "pnpm format:check && pnpm build && pnpm test && pnpm lint && pnpm test:exports && npm publish", "prepare": "husky || true" }, "lint-staged": { diff --git a/packages/aimock-pytest/pyproject.toml b/packages/aimock-pytest/pyproject.toml index 09e0894f..1cbd98c0 100644 --- a/packages/aimock-pytest/pyproject.toml +++ b/packages/aimock-pytest/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "aimock-pytest" -version = "0.4.0" +version = "0.5.0" description = "pytest fixtures for aimock — mock LLM APIs, multimedia, MCP, A2A, AG-UI, vector DBs, and more" readme = "README.md" requires-python = ">=3.10" @@ -12,7 +12,7 @@ license = "MIT" dependencies = ["requests>=2.28"] [project.optional-dependencies] -test = ["pytest>=7.0", "requests>=2.28"] +test = ["pytest>=7.0"] [project.entry-points."pytest11"] aimock = "aimock_pytest.plugin" diff --git a/packages/aimock-pytest/src/aimock_pytest/_server.py b/packages/aimock-pytest/src/aimock_pytest/_server.py index 946902de..07d98a09 100644 --- a/packages/aimock-pytest/src/aimock_pytest/_server.py +++ b/packages/aimock-pytest/src/aimock_pytest/_server.py @@ -182,6 +182,7 @@ def url(self) -> str: "systemMessage", "inputText", "toolCallId", + "toolResultContains", "toolName", "model", "responseFormat", diff --git a/packages/aimock-pytest/src/aimock_pytest/_version.py b/packages/aimock-pytest/src/aimock_pytest/_version.py index e1f62ee5..9c863968 100644 --- a/packages/aimock-pytest/src/aimock_pytest/_version.py +++ b/packages/aimock-pytest/src/aimock_pytest/_version.py @@ -8,4 +8,4 @@ control routes ship in the next release). Keep it tracking npm releases. """ -AIMOCK_VERSION = "1.35.1" +AIMOCK_VERSION = "1.37.0" diff --git a/skills/write-fixtures/SKILL.md b/skills/write-fixtures/SKILL.md index 8c3de1a4..c9c7e05a 100644 --- a/skills/write-fixtures/SKILL.md +++ b/skills/write-fixtures/SKILL.md @@ -19,25 +19,26 @@ aimock is a zero-dependency mock infrastructure for AI apps. Fixture-driven. Mul ## Match Field Reference -| Field | Type | Matches Against | -| ---------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `userMessage` | `string` | Substring of last `role: "user"` message text | -| `userMessage` | `RegExp` | Pattern test on last `role: "user"` message text | -| `systemMessage` | `string` | Substring of the concatenated text of every `role: "system"` message in the request. Use to gate a fixture on host-supplied context (persona, agent-context entries) so changes to that context cause the fixture to fall through instead of returning a stale baked response | -| `systemMessage` | `string[]` | Array of substrings — ALL must be present in the joined system text (AND semantics). Use when the gate must combine multiple non-adjacent tokens whose serialisation order isn't stable | -| `systemMessage` | `RegExp` | Pattern test on the concatenated system-message text | -| `inputText` | `string` | Substring of embedding input text (concatenated if multiple inputs) | -| `inputText` | `RegExp` | Pattern test on embedding input text | -| `toolName` | `string` | Exact match on any tool in request's `tools[]` array (by `function.name`) | -| `toolCallId` | `string` | Exact match on `tool_call_id` of last `role: "tool"` message | -| `model` | `string` | Exact match on `req.model` | -| `model` | `RegExp` | Pattern test on `req.model` | -| `responseFormat` | `string` | Exact match on `req.response_format.type` (`"json_object"`, `"json_schema"`) | -| `sequenceIndex` | `number` | Matches only when this fixture's match count equals the given index (0-based) | -| `turnIndex` | `number` | Stateless conversation-depth matching. Counts `role: "assistant"` messages in the request; matches when that count equals the value. `turnIndex: 0` = first turn (no prior assistant messages). Use instead of `sequenceIndex` for shared/deployed instances where stateful counters break under concurrency | -| `hasToolResult` | `boolean` | Stateless tool-message presence matching. `true` matches when any `role: "tool"` message exists in the request; `false` matches when none exist. Provider-consistent across all aimock handlers (OpenAI, Claude, Gemini, Bedrock, Ollama, Cohere) | -| `endpoint` | `string` | Restrict to endpoint type: `"chat"`, `"image"`, `"speech"`, `"transcription"`, `"video"`, `"embedding"` | -| `predicate` | `(req: ChatCompletionRequest) => boolean` | Custom function — full access to request | +| Field | Type | Matches Against | +| -------------------- | ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `userMessage` | `string` | Substring of last `role: "user"` message text | +| `userMessage` | `RegExp` | Pattern test on last `role: "user"` message text | +| `systemMessage` | `string` | Substring of the concatenated text of every `role: "system"` message in the request. Use to gate a fixture on host-supplied context (persona, agent-context entries) so changes to that context cause the fixture to fall through instead of returning a stale baked response | +| `systemMessage` | `string[]` | Array of substrings — ALL must be present in the joined system text (AND semantics). Use when the gate must combine multiple non-adjacent tokens whose serialisation order isn't stable | +| `systemMessage` | `RegExp` | Pattern test on the concatenated system-message text | +| `inputText` | `string` | Substring of embedding input text (concatenated if multiple inputs) | +| `inputText` | `RegExp` | Pattern test on embedding input text | +| `toolName` | `string` | Exact match on any tool in request's `tools[]` array (by `function.name`) | +| `toolCallId` | `string` | Exact match on `tool_call_id` of last `role: "tool"` message | +| `toolResultContains` | `string` | Substring of the last tool message's text content, gated on that message being the request's LAST message (same rule as `toolCallId`). Discriminates resume paths that share a `tool_call_id` and differ only inside the tool-result payload (e.g. approve `{"chosen_time": …}` vs cancel `{"cancelled": true}`) | +| `model` | `string` | Exact match on `req.model` | +| `model` | `RegExp` | Pattern test on `req.model` | +| `responseFormat` | `string` | Exact match on `req.response_format.type` (`"json_object"`, `"json_schema"`) | +| `sequenceIndex` | `number` | Matches only when this fixture's match count equals the given index (0-based) | +| `turnIndex` | `number` | Stateless conversation-depth matching. Counts `role: "assistant"` messages in the request; matches when that count equals the value. `turnIndex: 0` = first turn (no prior assistant messages). Use instead of `sequenceIndex` for shared/deployed instances where stateful counters break under concurrency | +| `hasToolResult` | `boolean` | Stateless tool-message presence matching. `true` matches when any `role: "tool"` message exists in the request; `false` matches when none exist. Provider-consistent across all aimock handlers (OpenAI, Claude, Gemini, Bedrock, Ollama, Cohere) | +| `endpoint` | `string` | Restrict to endpoint type: `"chat"`, `"image"`, `"speech"`, `"transcription"`, `"video"`, `"embedding"` | +| `predicate` | `(req: ChatCompletionRequest) => boolean` | Custom function — full access to request | **AND logic**: all specified fields must match. Empty match `{}` = catch-all. @@ -45,14 +46,15 @@ Multi-part content (e.g., `[{type: "text", text: "hello"}]`) is automatically ex ### When to Use Each Multi-turn Matching Approach -| Approach | Stateless? | Best For | -| --------------- | ---------- | --------------------------------------------------------------------------------------------------------- | -| `turnIndex` | Yes | Shared/deployed instances; matches on conversation depth (count of assistant messages in request) | -| `hasToolResult` | Yes | Simplest option for 2-step tool flows — boolean: are there tool results in the request? | -| `sequenceIndex` | No | Single-client unit tests with repeated identical requests (server-side counter, breaks under concurrency) | -| `toolCallId` | Yes | Matching specific tool result IDs in the conversation history | +| Approach | Stateless? | Best For | +| -------------------- | ---------- | --------------------------------------------------------------------------------------------------------- | +| `turnIndex` | Yes | Shared/deployed instances; matches on conversation depth (count of assistant messages in request) | +| `hasToolResult` | Yes | Simplest option for 2-step tool flows — boolean: are there tool results in the request? | +| `sequenceIndex` | No | Single-client unit tests with repeated identical requests (server-side counter, breaks under concurrency) | +| `toolCallId` | Yes | Matching specific tool result IDs in the conversation history | +| `toolResultContains` | Yes | Same tool call id, different outcomes — match on the tool-result payload (approve vs cancel legs) | -**Prefer stateless approaches** (`turnIndex`, `hasToolResult`) for shared aimock instances (deployed via Docker, used by multiple test runners). Use `sequenceIndex` only in isolated single-client unit tests where the counter won't be corrupted by concurrent requests. +**Prefer stateless approaches** (`turnIndex`, `hasToolResult`, `toolResultContains`) for shared aimock instances (deployed via Docker, used by multiple test runners). Use `sequenceIndex` only in isolated single-client unit tests where the counter won't be corrupted by concurrent requests. ### Multi-turn fixture examples @@ -64,6 +66,11 @@ Multi-part content (e.g., `[{type: "text", text: "hello"}]`) is automatically ex // Same thing with hasToolResult (simpler for 2-step) {"match": {"userMessage": "trip to mars", "hasToolResult": false}, "response": {"toolCalls": [{"id": "call_001", "name": "generate_steps", "arguments": "{}"}]}} {"match": {"userMessage": "trip to mars", "hasToolResult": true}, "response": {"content": "Great choices!"}} + +// HITL suspend tool where approve and cancel resume with the SAME tool call id — +// discriminate on the tool-result payload; put the cancel leg first (first match wins) +{"match": {"toolCallId": "call_001", "toolResultContains": "\"cancelled\""}, "response": {"content": "No problem — nothing was booked."}} +{"match": {"toolCallId": "call_001"}, "response": {"content": "Booked: Monday 9:00 AM confirmed."}} ``` ## Response Types @@ -255,7 +262,7 @@ mock.on( mock.on({ userMessage: "status", sequenceIndex: 1 }, { content: "All systems operational." }); ``` -Match counts are tracked per fixture group and reset with `reset()` or `resetMatchCounts()`. +Match counts are tracked per fixture group. Use `resetMatchCounts()` between tests to reset counts while keeping loaded fixtures. `reset()` also clears the fixture pool, so avoid it between tests that share a loaded fixture set. ### Streaming physics (realistic timing) @@ -500,7 +507,7 @@ These fields map correctly across all provider formats — for example, `finishR 10. **Embeddings auto-generate if no fixture matches** — deterministic vectors are generated from the input text hash. You don't need a catch-all for embedding requests. -11. **Sequential response counts are tracked per fixture** — counts reset with `reset()` or `resetMatchCounts()`. The count increments after each match of that fixture group (all fixtures sharing the same non-`sequenceIndex` match fields). +11. **Sequential response counts are tracked per fixture** — use `resetMatchCounts()` between tests to reset counts while keeping loaded fixtures; `reset()` also clears the fixture pool, so don't use it between tests that share a loaded fixture set. The count increments after each match of that fixture group (all fixtures sharing the same non-`sequenceIndex` match fields). 12. **Bedrock uses Anthropic Messages format internally** — the adapter normalizes Bedrock requests to `ChatCompletionRequest`, so the same fixtures work. Bedrock supports both non-streaming (`/invoke`, `/converse`) and streaming (`/invoke-with-response-stream`, `/converse-stream`) endpoints. @@ -549,7 +556,7 @@ await suite.start(); // suite.llm — the LLMock instance // suite.url — base URL -afterEach(() => suite.reset()); // resets everything +afterEach(() => suite.llm.resetMatchCounts()); // reset sequence counts, keep fixtures afterAll(() => suite.stop()); ``` @@ -684,8 +691,8 @@ mock.loadFixtureDir("./fixtures"); await mock.start(); process.env.OPENAI_BASE_URL = `${mock.url}/v1`; -// Per-test cleanup -afterEach(() => mock.reset()); // clears fixtures AND journal +// Per-test cleanup — reset sequence match counts, keep the loaded fixtures +afterEach(() => mock.resetMatchCounts()); // Teardown afterAll(async () => await mock.stop()); @@ -736,6 +743,8 @@ const mock = await LLMock.create({ port: 0 }); // creates + starts in one call | `url` / `baseUrl` | Server URL (throws if not started) | | `port` | Server port number | +Between tests that share a loaded fixture set, use `resetMatchCounts()` (not `reset()`, which also clears fixtures). For a `MockSuite`, call `suite.llm.resetMatchCounts()` — the suite itself has no `resetMatchCounts()`. + Sequential responses use `on()` with `sequenceIndex` in the match — there is no dedicated convenience method. ## Record-and-Replay (VCR Mode) diff --git a/src/__tests__/fixture-loader.test.ts b/src/__tests__/fixture-loader.test.ts index ca4719c1..1617eda2 100644 --- a/src/__tests__/fixture-loader.test.ts +++ b/src/__tests__/fixture-loader.test.ts @@ -316,6 +316,32 @@ describe("loadFixtureFile", () => { expect(fixtures[0].disconnectAfterMs).toBeUndefined(); }); + it("loads sibling fixtures when one entry has a non-array interChunkDelaysMs", () => { + // Untrusted fixture JSON: interChunkDelaysMs is typed number[] but may be + // null/missing/non-array. Without an Array.isArray guard, .filter throws a + // TypeError inside entryToFixture, aborting the entire file's load and + // silently dropping every fixture in it. + const filePath = writeJson(tmpDir, "bad-timings.json", { + fixtures: [ + { + match: { userMessage: "malformed" }, + response: { content: "bad" }, + recordedTimings: { ttftMs: 0, interChunkDelaysMs: null, totalDurationMs: 0 }, + }, + { + match: { userMessage: "valid" }, + response: { content: "good" }, + }, + ], + }); + + const fixtures = loadFixtureFile(filePath); + expect(fixtures).toHaveLength(2); + expect(fixtures[1].match.userMessage).toBe("valid"); + // The malformed entry is sanitized: non-array interChunkDelaysMs -> []. + expect(fixtures[0].recordedTimings?.interChunkDelaysMs).toEqual([]); + }); + it("warns and returns empty array for invalid JSON", () => { const filePath = join(tmpDir, "bad.json"); writeFileSync(filePath, "{ not valid json", "utf-8"); @@ -1233,6 +1259,34 @@ describe("validateFixtures", () => { expect(results.filter((r) => r.message.includes("hasToolResult"))).toHaveLength(0); }); + // --- match.toolResultContains type checks --- + + it("error: toolResultContains is a number", () => { + const fixtures = [ + makeFixture({ match: { userMessage: "test", toolResultContains: 42 as never } }), + ]; + const results = validateFixtures(fixtures); + expect( + results.some((r) => r.severity === "error" && r.message.includes("toolResultContains")), + ).toBe(true); + }); + + it("error: toolResultContains is an empty string", () => { + const fixtures = [makeFixture({ match: { userMessage: "test", toolResultContains: "" } })]; + const results = validateFixtures(fixtures); + expect( + results.some((r) => r.severity === "error" && r.message.includes("toolResultContains")), + ).toBe(true); + }); + + it("no error: toolResultContains is a non-empty string", () => { + const fixtures = [ + makeFixture({ match: { userMessage: "test", toolResultContains: "cancelled" } }), + ]; + const results = validateFixtures(fixtures); + expect(results.filter((r) => r.message.includes("toolResultContains"))).toHaveLength(0); + }); + // --- match.systemMessage type checks --- it("error: systemMessage is a number", () => { @@ -1348,6 +1402,18 @@ describe("validateFixtures", () => { expect(duplicateWarnings).toHaveLength(0); }); + it("no warning: same userMessage but different toolResultContains", () => { + const fixtures = [ + makeFixture({ match: { userMessage: "hello", toolResultContains: "cancelled" } }), + makeFixture({ match: { userMessage: "hello", toolResultContains: "chosen_" } }), + ]; + const results = validateFixtures(fixtures); + const duplicateWarnings = results.filter( + (r) => r.severity === "warning" && r.message.includes("duplicate"), + ); + expect(duplicateWarnings).toHaveLength(0); + }); + it("no warning: same userMessage but different sequenceIndex", () => { const fixtures = [ makeFixture({ match: { userMessage: "hello", sequenceIndex: 0 } }), @@ -1360,6 +1426,90 @@ describe("validateFixtures", () => { expect(duplicateWarnings).toHaveLength(0); }); + it("no warning: same userMessage but different toolCallId", () => { + const fixtures = [ + makeFixture({ match: { userMessage: "hello", toolCallId: "call_a" } }), + makeFixture({ match: { userMessage: "hello", toolCallId: "call_b" } }), + ]; + const results = validateFixtures(fixtures); + const duplicateWarnings = results.filter( + (r) => r.severity === "warning" && r.message.includes("duplicate"), + ); + expect(duplicateWarnings).toHaveLength(0); + }); + + it("no warning: same userMessage but different systemMessage", () => { + const fixtures = [ + makeFixture({ match: { userMessage: "hello", systemMessage: "persona A" } }), + makeFixture({ match: { userMessage: "hello", systemMessage: "persona B" } }), + ]; + const results = validateFixtures(fixtures); + const duplicateWarnings = results.filter( + (r) => r.severity === "warning" && r.message.includes("duplicate"), + ); + expect(duplicateWarnings).toHaveLength(0); + }); + + it("no warning: same userMessage but different model", () => { + const fixtures = [ + makeFixture({ match: { userMessage: "hello", model: "gpt-4o" } }), + makeFixture({ match: { userMessage: "hello", model: "claude-opus-4" } }), + ]; + const results = validateFixtures(fixtures); + const duplicateWarnings = results.filter( + (r) => r.severity === "warning" && r.message.includes("duplicate"), + ); + expect(duplicateWarnings).toHaveLength(0); + }); + + it("no warning: same userMessage but different toolName", () => { + const fixtures = [ + makeFixture({ match: { userMessage: "hello", toolName: "search" } }), + makeFixture({ match: { userMessage: "hello", toolName: "fetch" } }), + ]; + const results = validateFixtures(fixtures); + const duplicateWarnings = results.filter( + (r) => r.severity === "warning" && r.message.includes("duplicate"), + ); + expect(duplicateWarnings).toHaveLength(0); + }); + + it("no warning: same userMessage but different responseFormat", () => { + const fixtures = [ + makeFixture({ match: { userMessage: "hello", responseFormat: "text" } }), + makeFixture({ match: { userMessage: "hello", responseFormat: "json_object" } }), + ]; + const results = validateFixtures(fixtures); + const duplicateWarnings = results.filter( + (r) => r.severity === "warning" && r.message.includes("duplicate"), + ); + expect(duplicateWarnings).toHaveLength(0); + }); + + it("no warning: same userMessage but different endpoint", () => { + const fixtures = [ + makeFixture({ match: { userMessage: "hello", endpoint: "chat" } }), + makeFixture({ match: { userMessage: "hello", endpoint: "video" } }), + ]; + const results = validateFixtures(fixtures); + const duplicateWarnings = results.filter( + (r) => r.severity === "warning" && r.message.includes("duplicate"), + ); + expect(duplicateWarnings).toHaveLength(0); + }); + + it("no warning: same userMessage but different inputText", () => { + const fixtures = [ + makeFixture({ match: { userMessage: "hello", inputText: "embed A" } }), + makeFixture({ match: { userMessage: "hello", inputText: "embed B" } }), + ]; + const results = validateFixtures(fixtures); + const duplicateWarnings = results.filter( + (r) => r.severity === "warning" && r.message.includes("duplicate"), + ); + expect(duplicateWarnings).toHaveLength(0); + }); + it("warning: same userMessage with identical turnIndex/hasToolResult/sequenceIndex", () => { const fixtures = [ makeFixture({ match: { userMessage: "hello", turnIndex: 1, hasToolResult: true } }), @@ -1868,6 +2018,15 @@ describe("auto-stringify JSON objects in fixture entries", () => { expect(fixture.match.systemMessage).toBe("name=Atai"); }); + it("passes toolResultContains through entryToFixture", () => { + const entry: FixtureFileEntry = { + match: { toolCallId: "call_x", toolResultContains: '"cancelled"' }, + response: { content: "ok" }, + }; + const fixture = entryToFixture(entry); + expect(fixture.match.toolResultContains).toBe('"cancelled"'); + }); + it("stringifies nested objects in arguments", () => { const entry: FixtureFileEntry = { match: { userMessage: "test" }, diff --git a/src/__tests__/router.test.ts b/src/__tests__/router.test.ts index 50885730..8ac4396d 100644 --- a/src/__tests__/router.test.ts +++ b/src/__tests__/router.test.ts @@ -585,6 +585,127 @@ describe("matchFixture — toolCallId", () => { }); }); +// --------------------------------------------------------------------------- +// matchFixture — toolResultContains +// --------------------------------------------------------------------------- + +describe("matchFixture — toolResultContains", () => { + it("discriminates approve vs cancel legs that share a toolCallId", () => { + // The motivating case: a human-in-the-loop suspend tool resumes with the + // SAME tool_call_id for both outcomes; only the tool-result JSON differs. + const cancelled = makeFixture( + { toolCallId: "call_schedule_001", toolResultContains: '"cancelled"' }, + { content: "No problem — nothing was booked." }, + ); + const confirmed = makeFixture( + { toolCallId: "call_schedule_001", toolResultContains: '"chosen_' }, + { content: "Booked: Monday 9:00 AM confirmed." }, + ); + const base = [ + { role: "user" as const, content: "schedule a meeting" }, + { + role: "assistant" as const, + content: null, + tool_calls: [ + { + id: "call_schedule_001", + type: "function" as const, + function: { name: "schedule_meeting", arguments: "{}" }, + }, + ], + }, + ]; + const cancelReq = makeReq({ + messages: [ + ...base, + { role: "tool", content: '{"cancelled": true}', tool_call_id: "call_schedule_001" }, + ], + }); + const pickReq = makeReq({ + messages: [ + ...base, + { + role: "tool", + content: '{"chosen_time": "2026-07-20T09:00:00Z", "chosen_label": "Monday 9:00 AM"}', + tool_call_id: "call_schedule_001", + }, + ], + }); + const fixtures = [cancelled, confirmed]; + expect(matchFixture(fixtures, cancelReq)).toBe(cancelled); + expect(matchFixture(fixtures, pickReq)).toBe(confirmed); + }); + + it("matches when the last tool message contains the substring", () => { + const fixture = makeFixture({ toolResultContains: "cancelled" }); + const req = makeReq({ + messages: [ + { role: "user", content: "do thing" }, + { role: "tool", content: '{"cancelled": true}', tool_call_id: "call_x" }, + ], + }); + expect(matchFixture([fixture], req)).toBe(fixture); + }); + + it("does not match when the substring is absent", () => { + const fixture = makeFixture({ toolResultContains: "cancelled" }); + const req = makeReq({ + messages: [ + { role: "user", content: "do thing" }, + { role: "tool", content: '{"chosen_time": "9am"}', tool_call_id: "call_x" }, + ], + }); + expect(matchFixture([fixture], req)).toBeNull(); + }); + + it("does not match a request with no tool message", () => { + const fixture = makeFixture({ toolResultContains: "cancelled" }); + const req = makeReq({ messages: [{ role: "user", content: "cancelled my plans" }] }); + expect(matchFixture([fixture], req)).toBeNull(); + }); + + it("does not match when a newer user turn follows the tool message", () => { + // Same last-message rule as toolCallId: a stale tool result in history + // must not shadow matchers for the new turn. + const fixture = makeFixture({ toolResultContains: "cancelled" }); + const req = makeReq({ + messages: [ + { role: "user", content: "do thing" }, + { role: "tool", content: '{"cancelled": true}', tool_call_id: "call_x" }, + { role: "assistant", content: "Cancelled." }, + { role: "user", content: "something else" }, + ], + }); + expect(matchFixture([fixture], req)).toBeNull(); + }); + + it("matches array-of-parts tool content", () => { + const fixture = makeFixture({ toolResultContains: "cancelled" }); + const req = makeReq({ + messages: [ + { role: "user", content: "do thing" }, + { + role: "tool", + content: [{ type: "text", text: '{"cancelled": true}' }], + tool_call_id: "call_x", + }, + ], + }); + expect(matchFixture([fixture], req)).toBe(fixture); + }); + + it("does not match tool content with no extractable text", () => { + const fixture = makeFixture({ toolResultContains: "cancelled" }); + const req = makeReq({ + messages: [ + { role: "user", content: "do thing" }, + { role: "tool", content: null, tool_call_id: "call_x" }, + ], + }); + expect(matchFixture([fixture], req)).toBeNull(); + }); +}); + // --------------------------------------------------------------------------- // matchFixture — toolName // --------------------------------------------------------------------------- diff --git a/src/fixture-loader.ts b/src/fixture-loader.ts index 9d911fcb..4fe26366 100644 Binary files a/src/fixture-loader.ts and b/src/fixture-loader.ts differ diff --git a/src/router.ts b/src/router.ts index fe6ec21b..c74f465c 100644 --- a/src/router.ts +++ b/src/router.ts @@ -347,6 +347,19 @@ export function matchFixtureDiagnostic( if (!last || last.role !== "tool" || last.tool_call_id !== match.toolCallId) continue; } + // toolResultContains — substring gate on the LAST message's text content + // when that message is a tool result. Same last-message rule as toolCallId + // (a tool-result fixture answers the model's next call after a tool round), + // but discriminates on the result PAYLOAD instead of the call id — needed + // when approve/cancel resumes share the same toolCallId and differ only + // inside the tool-result JSON. + if (match.toolResultContains !== undefined) { + const last = effective.messages[effective.messages.length - 1]; + if (!last || last.role !== "tool") continue; + const text = getTextContent(last.content); + if (text === null || !text.includes(match.toolResultContains)) continue; + } + // toolName — match against any tool definition by function.name if (match.toolName !== undefined) { const tools = effective.tools ?? []; diff --git a/src/types.ts b/src/types.ts index b7a0aebe..51301447 100644 --- a/src/types.ts +++ b/src/types.ts @@ -92,6 +92,14 @@ export interface FixtureMatch { systemMessage?: string | string[] | RegExp; inputText?: string | RegExp; toolCallId?: string; + /** + * Substring matched against the text content of the LAST message when that + * message is a `tool` result (same last-message rule as `toolCallId`). + * Discriminates fixtures whose requests differ ONLY inside the tool-result + * payload — e.g. a human-in-the-loop tool where approve and cancel resume + * with the same toolCallId but different result JSON. + */ + toolResultContains?: string; toolName?: string; model?: string | RegExp; responseFormat?: string; @@ -541,6 +549,11 @@ export interface FixtureFileEntry { systemMessage?: string | string[]; inputText?: string; toolCallId?: string; + /** + * Substring matched against the last tool-result message's text content. + * Mirrors the runtime FixtureMatch.toolResultContains. + */ + toolResultContains?: string; toolName?: string; model?: string; responseFormat?: string;

    • Different behavior before vs. after tool execution (tool-call round trip) toolCallId - Matches the tool_call_id of the last role: "tool" - message. Turn 1 has no tool message; turn 2 does. + Matches when the request’s overall last message is a + role: "tool" result with that tool_call_id. Turn 1 has no + tool result last; turn 2 does. +
    + Two tool-result outcomes that share a tool_call_id and differ only + inside the result payload (e.g. approve vs. cancel) + toolResultContains + Substring match on the last role: "tool" result’s text. Use when + toolCallId alone can’t tell the outcomes apart. Stateless.