fix: release-gate exports check, fixture-load guard, and E2E doc lifecycle#300
Closed
jpr5 wants to merge 3 commits into
Closed
fix: release-gate exports check, fixture-load guard, and E2E doc lifecycle#300jpr5 wants to merge 3 commits into
jpr5 wants to merge 3 commits into
Conversation
The release script published without verifying formatting or the package export map, so unformatted code or a broken export map could reach npm (recurring failure, see CHANGELOG #263). Add format:check before build (cheap, fail fast) and test:exports after build (needs the built dist), with npm publish last.
Untrusted fixture JSON may set interChunkDelaysMs to null/missing/non-array. The sanitizer called .filter with no Array.isArray guard, throwing a TypeError inside entryToFixture that aborted the entire fixture file's load and silently dropped every fixture in it. Coerce non-array values to [] like sibling guards.
The E2E setup examples loaded fixtures once then called full reset() in afterEach, which wipes the in-memory fixture pool and 404s every test after the first. Switch between-test cleanup to resetMatchCounts() (suite.llm.resetMatchCounts() for MockSuite) and correct the prose so it leads with resetMatchCounts() and warns reset() also clears fixtures.
commit: |
Contributor
Author
jpr5
added a commit
that referenced
this pull request
Jul 16, 2026
…299) ## Why The showcase mastra interrupt demos use a native suspend tool where pick and cancel resume with the SAME tool_call_id; the requests differ only inside the tool-result message content (`{"chosen_time": ...}` vs `{"cancelled": true}`). Every existing JSON-expressible gate is identical between the two requests, so first-match-wins replays the pick confirmation after the user cancelled. `predicate` cannot help: the JSON fixture loader has no way to express it. ## What New `match.toolResultContains: string` gate: passes when the request's LAST message is a `role: "tool"` result whose text content contains the substring. Mirrors the existing `toolCallId` last-message rule so the two stay consistent. - `src/router.ts`: gate placed with the other shape predicates, right after `toolCallId` - `src/types.ts`: `FixtureMatch` + `FixtureFileEntry.match` - `src/fixture-loader.ts`: `entryToFixture` carry-through; validation (string, non-empty — empty substring would silently act as a catch-all); userMessage dedup key; catch-all discriminator set - `packages/aimock-pytest`: `toolResultContains` added to `_MATCH_LEVEL_OPT_KEYS` (0.5.0) - docs: fixtures + multi-turn pages, README, write-fixtures skill - tests: approve/cancel discrimination sharing one toolCallId; substring present/absent; no-tool-message request never matches; stale tool result followed by a new user turn never matches; array-of-parts content; null content; loader validation + dedup + carry-through ## Version 1.37.0 (changelog entry added with this PR's number in a follow-up commit on this branch). --- ### Maintainer note — additional fixes folded in during review Beyond the `toolResultContains` gate, this branch now also carries (all under the single 1.37.0 release): - **fix:** completed the duplicate-userMessage dedup key so legitimately-distinct fixtures no longer emit spurious shadow warnings (kind-aware matcher serialization; `predicate` keyed by identity). - **fix:** guarded `recordedTimings.interChunkDelaysMs` against non-array values — a malformed entry no longer throws and silently drops every fixture in its file. - **fix:** bumped the `aimock-pytest` default server pin to 1.37.0 so the auto-download path actually supports the new kwarg the client forwards. - **ci:** the `release` script now runs `format:check` + `test:exports` before publish. - **docs:** corrected the match-field / dedup / `toolCallId` semantics docs, and fixed the write-fixtures E2E lifecycle to use `resetMatchCounts()` instead of a full `reset()` between tests. (These consolidate PR #300, now closed, into this one release. CHANGELOG updated accordingly.)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Three pre-existing issues surfaced while reviewing #299 (the
toolResultContainsgate). None block that PR; they're worth fixing on their own. Prioritized from the review backlog as the items with real teeth (a publish-safety gap, a fixture-load footgun, and a broken canonical test example).What
1. Release gate now validates exports + formatting (
ci:)releaseranbuild && test && lint && publish— it skippedtest:exports(publint + attw) andformat:check, so a broken export map could publish (this class of failure recurred, see CHANGELOG #263). Now:format:check && build && test && lint && test:exports && publish(format up front, exports after build since it needsdist/).2. A malformed fixture no longer aborts the whole file (
fix:)recordedTimings.interChunkDelaysMswas.filter-ed with noArray.isArrayguard (its siblingsttftMs/totalDurationMsare guarded). SinceloadFixtureFilemaps entries with no try/catch, a single fixture with a non-array value threw aTypeErrorthat took down every fixture in the file — silently, via the loader's warn-and-return-empty wrapper. Now coerced to[].Red-green on the real loader:
TypeError: Cannot read properties of null (reading 'filter')— sibling fixtures fail to load.3. Fixed the broken E2E lifecycle in the write-fixtures skill (
docs:)The canonical setup examples loaded fixtures once, then called full
reset()inafterEach— which clears the fixture pool, so every test after the first 404s. Switched to the narrowresetMatchCounts()(andsuite.llm.resetMatchCounts()for the suite —MockSuitehas noresetMatchCounts), and reworded the two prose spots that steered readers toreset()between tests.A full sweep of every doc surface confirmed the anti-pattern was confined to
skills/write-fixtures/SKILL.md— thedocs/*.htmlpages are correct (docs/test-pluginsanddocs/control-apiare the right reference and already warn about it), and pytest fixtures are function-scoped so they're immune.Test
format:check,lint,tsc --noEmit,test:exports(all 🟢),build, and the full suite (4446 tests) all pass locally.