Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@
"test:exports": "publint && attw --pack .",
"lint": "eslint .",
"format:check": "prettier --check .",
"release": "pnpm build && pnpm test && pnpm lint && npm publish",
"release": "pnpm format:check && pnpm build && pnpm test && pnpm lint && pnpm test:exports && npm publish",
"prepare": "husky || true"
},
"lint-staged": {
Expand Down
12 changes: 7 additions & 5 deletions skills/write-fixtures/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,7 @@ mock.on(
mock.on({ userMessage: "status", sequenceIndex: 1 }, { content: "All systems operational." });
```

Match counts are tracked per fixture group and reset with `reset()` or `resetMatchCounts()`.
Match counts are tracked per fixture group. Use `resetMatchCounts()` between tests to reset counts while keeping loaded fixtures. `reset()` also clears the fixture pool, so avoid it between tests that share a loaded fixture set.

### Streaming physics (realistic timing)

Expand Down Expand Up @@ -500,7 +500,7 @@ These fields map correctly across all provider formats — for example, `finishR

10. **Embeddings auto-generate if no fixture matches** — deterministic vectors are generated from the input text hash. You don't need a catch-all for embedding requests.

11. **Sequential response counts are tracked per fixture** — counts reset with `reset()` or `resetMatchCounts()`. The count increments after each match of that fixture group (all fixtures sharing the same non-`sequenceIndex` match fields).
11. **Sequential response counts are tracked per fixture** — use `resetMatchCounts()` between tests to reset counts while keeping loaded fixtures; `reset()` also clears the fixture pool, so don't use it between tests that share a loaded fixture set. The count increments after each match of that fixture group (all fixtures sharing the same non-`sequenceIndex` match fields).

12. **Bedrock uses Anthropic Messages format internally** — the adapter normalizes Bedrock requests to `ChatCompletionRequest`, so the same fixtures work. Bedrock supports both non-streaming (`/invoke`, `/converse`) and streaming (`/invoke-with-response-stream`, `/converse-stream`) endpoints.

Expand Down Expand Up @@ -549,7 +549,7 @@ await suite.start();
// suite.llm — the LLMock instance
// suite.url — base URL

afterEach(() => suite.reset()); // resets everything
afterEach(() => suite.llm.resetMatchCounts()); // reset sequence counts, keep fixtures
afterAll(() => suite.stop());
```

Expand Down Expand Up @@ -684,8 +684,8 @@ mock.loadFixtureDir("./fixtures");
await mock.start();
process.env.OPENAI_BASE_URL = `${mock.url}/v1`;

// Per-test cleanup
afterEach(() => mock.reset()); // clears fixtures AND journal
// Per-test cleanup — reset sequence match counts, keep the loaded fixtures
afterEach(() => mock.resetMatchCounts());

// Teardown
afterAll(async () => await mock.stop());
Expand Down Expand Up @@ -736,6 +736,8 @@ const mock = await LLMock.create({ port: 0 }); // creates + starts in one call
| `url` / `baseUrl` | Server URL (throws if not started) |
| `port` | Server port number |

Between tests that share a loaded fixture set, use `resetMatchCounts()` (not `reset()`, which also clears fixtures). For a `MockSuite`, call `suite.llm.resetMatchCounts()` — the suite itself has no `resetMatchCounts()`.

Sequential responses use `on()` with `sequenceIndex` in the match — there is no dedicated convenience method.

## Record-and-Replay (VCR Mode)
Expand Down
26 changes: 26 additions & 0 deletions src/__tests__/fixture-loader.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -316,6 +316,32 @@ describe("loadFixtureFile", () => {
expect(fixtures[0].disconnectAfterMs).toBeUndefined();
});

it("loads sibling fixtures when one entry has a non-array interChunkDelaysMs", () => {
// Untrusted fixture JSON: interChunkDelaysMs is typed number[] but may be
// null/missing/non-array. Without an Array.isArray guard, .filter throws a
// TypeError inside entryToFixture, aborting the entire file's load and
// silently dropping every fixture in it.
const filePath = writeJson(tmpDir, "bad-timings.json", {
fixtures: [
{
match: { userMessage: "malformed" },
response: { content: "bad" },
recordedTimings: { ttftMs: 0, interChunkDelaysMs: null, totalDurationMs: 0 },
},
{
match: { userMessage: "valid" },
response: { content: "good" },
},
],
});

const fixtures = loadFixtureFile(filePath);
expect(fixtures).toHaveLength(2);
expect(fixtures[1].match.userMessage).toBe("valid");
// The malformed entry is sanitized: non-array interChunkDelaysMs -> [].
expect(fixtures[0].recordedTimings?.interChunkDelaysMs).toEqual([]);
});

it("warns and returns empty array for invalid JSON", () => {
const filePath = join(tmpDir, "bad.json");
writeFileSync(filePath, "{ not valid json", "utf-8");
Expand Down
4 changes: 3 additions & 1 deletion src/fixture-loader.ts
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,9 @@ export function entryToFixture(entry: FixtureFileEntry, logger?: Logger): Fixtur
if (fixture.recordedTimings) {
const rt = fixture.recordedTimings;
if (!Number.isFinite(rt.ttftMs) || rt.ttftMs < 0) rt.ttftMs = 0;
rt.interChunkDelaysMs = rt.interChunkDelaysMs.filter((d) => Number.isFinite(d) && d >= 0);
rt.interChunkDelaysMs = Array.isArray(rt.interChunkDelaysMs)
? rt.interChunkDelaysMs.filter((d) => Number.isFinite(d) && d >= 0)
: [];
if (!Number.isFinite(rt.totalDurationMs) || rt.totalDurationMs < 0) rt.totalDurationMs = 0;
}

Expand Down
Loading