Repository navigation
ci: add mutation testing with Stryker - #18
Merged
Merged
Conversation
Add Stryker with the vitest runner, `npm run mutate`, and a `mutate` workflow (weekly and workflow_dispatch, not on PRs) that uploads the HTML report as an artifact. Mutate src/index.ts, src/path-parts.ts and src/transform.ts: the parser was split into modules after the plan said "only src/index.ts". errors.ts is left out (constructors only). Stryker's tsconfig preprocessor calls ts.parseConfigFileTextToJson, which TypeScript 7 doesn't ship, so every run crashed. It only rewrites extends/references paths, which our tsconfig doesn't have; pointing tsconfigFile at a missing file skips it. The first five runs scored 95.7-97.0% (fast-check data differs per run), so thresholds.break is 93.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #18 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 4 4
Lines 134 134
Branches 45 45
=========================================
Hits 134 134 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The kills that vary between runs all come from the timing tests that parse 200 000-character paths (500 ms limit), which sometimes fail under Stryker's parallel load. fast-check didn't kill any of them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Roadmap 1.9. Adds mutation testing.
ci:doesn't release, and nothing in the package changes.Changes
@stryker-mutator/coreand@stryker-mutator/vitest-runner(10.1.1). No install scripts, soallowScriptsis unchanged. No@stryker-mutator/typescript-checker: it needs the TypeScript compiler API, which TS 7 doesn't ship.stryker.config.mjs(.mjsrather than JSON so the workaround below can have a comment):src/index.ts,src/path-parts.tsandsrc/transform.ts. The plan said onlysrc/index.ts, but it was written before the parser was split into modules.src/errors.tsis left out (constructors only).reports/mutation/mutation.html), clear-text, progressthresholds: {high: 97, low: 95, break: 93}npm run mutate(stryker run).reports/and.stryker-tmp/are gitignored..github/workflows/mutate.yml: runs weekly (Mon 04:17 UTC) and onworkflow_dispatch, not on PRs. Node 24,npm ci --ignore-scripts, uploads the HTML report as themutation-reportartifact (30 days), also when the score is belowbreak. Same pinned action SHAs asvalidate.yml. actionlint passes locally.TypeScript 7 workaround
Every run crashed with
TypeError: ts.parseConfigFileTextToJson is not a function. Before copying the project to its sandbox, Stryker'sTSConfigPreprocessoruses the TypeScript JS API to rewriteextends/referencespaths intsconfig.json. TS 7 has no JS API. Our tsconfig has neither field, and the preprocessor skips the step when the configuredtsconfigFileisn't in the project, so the config setstsconfigFile: 'tsconfig.stryker-skip.json'(a file that doesn't exist).First runs
A run takes ~33 s locally (16 threads). 303 mutants across five runs:
Four more runs with the JSON reporter scored 95.05–96.04%. The JSON report says which test killed each mutant: every kill that varies between runs comes from the timing tests ("parses a 200 000 character path of
a[b]/a[0]bquickly", 500 ms limit), which sometimes go over the limit while Stryker runs 16 workers in parallel. fast-check didn't kill any of the varying mutants.break: 93leaves room for this noise, so the weekly job doesn't fail at random. Checked: withbreak: 99the run exits 1 and still writes the report.Surviving mutants
Left for a follow-up
test:PR, as planned.Survived all 4 JSON runs:
src/transform.ts:64:value === 'true'→falseorvalue === "". No test checks that&a=truegivestrue. Real gap.src/index.ts:176-177:writable/enumerable: true→falsein thedefinePropertyfor prototype names. These entries are removed before returning, so probably equivalent.src/index.ts:367:[]→["Stryker was here"]. Needs a closer look.src/index.ts:377:pathPart.type === 'object'→true. Equivalent: array path parts are digits or empty, never onObject.prototype.src/index.ts:407:isRemoved && nextPathValue === undefined→isRemoved && true. Needs a closer look.src/path-parts.ts:47:if (open > lastClose) return -1→if (false). Needs a closer look.src/path-parts.ts:119:close < idx→close <= idx. Probably equivalent.Only ever killed by a timing test (1–2 of 4 runs):
index.ts:166(if (!isPrototypeKey(key))→false, and its block emptied; sends every key through the slowerdefineProperty, so partly a real perf kill),path-parts.ts:47(>→>=),path-parts.ts:49(<→<=),path-parts.ts:117(<→<=),path-parts.ts:122(>→>=). The loop-bound ones add at most one loop step, so those kills are noise.npm audit
npm auditreports 5 moderate findings, all in Stryker's dev-only dependency tree:minimatch10.2.6 (GHSA-g3hv-75wg-m249, glob pattern amplification; the patterns come from our config)qs6.15.1 viatyped-rest-client(used by the dashboard reporter, which we don't use)typed-rest-clientpinsqsexactly, so only anoverridesentry could fix it.mainhas 0 findings. CI doesn't runnpm audit, and nothing ships in the package.npm run validatepasses (knip is happy withstryker.config.mjs; I dropped a JSDoc type import from@stryker-mutator/apithat knip reported as an unlisted dependency). The workflow can only be started withworkflow_dispatchonce it's onmain.