Skip to content

ci: add mutation testing with Stryker - #18

Merged
milamer merged 2 commits into
mainfrom
ci/stryker
Nov 9, 2026
Merged

milamer merged 2 commits into
mainfrom
ci/stryker

Conversation

@milamer

@milamer milamer commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

Roadmap 1.9. Adds mutation testing. ci: doesn't release, and nothing in the package changes.

Changes

  • Dev dependencies @stryker-mutator/core and @stryker-mutator/vitest-runner (10.1.1). No install scripts, so allowScripts is unchanged. No @stryker-mutator/typescript-checker: it needs the TypeScript compiler API, which TS 7 doesn't ship.
  • stryker.config.mjs (.mjs rather than JSON so the workaround below can have a comment):
    • mutates src/index.ts, src/path-parts.ts and src/transform.ts. The plan said only src/index.ts, but it was written before the parser was split into modules. src/errors.ts is left out (constructors only).
    • reporters: html (reports/mutation/mutation.html), clear-text, progress
    • thresholds: {high: 97, low: 95, break: 93}
  • npm run mutate (stryker run). reports/ and .stryker-tmp/ are gitignored.
  • .github/workflows/mutate.yml: runs weekly (Mon 04:17 UTC) and on workflow_dispatch, not on PRs. Node 24, npm ci --ignore-scripts, uploads the HTML report as the mutation-report artifact (30 days), also when the score is below break. Same pinned action SHAs as validate.yml. actionlint passes locally.

TypeScript 7 workaround

Every run crashed with TypeError: ts.parseConfigFileTextToJson is not a function. Before copying the project to its sandbox, Stryker's TSConfigPreprocessor uses the TypeScript JS API to rewrite extends/references paths in tsconfig.json. TS 7 has no JS API. Our tsconfig has neither field, and the preprocessor skips the step when the configured tsconfigFile isn't in the project, so the config sets tsconfigFile: 'tsconfig.stryker-skip.json' (a file that doesn't exist).

First runs

A run takes ~33 s locally (16 threads). 303 mutants across five runs:

run score killed timeout survived
1 97.03% 287 7 9
2 96.70% 286 7 10
3 97.03% 290 4 9
4 95.71% 282 8 13
5 96.37% 284 8 11

Four more runs with the JSON reporter scored 95.05–96.04%. The JSON report says which test killed each mutant: every kill that varies between runs comes from the timing tests ("parses a 200 000 character path of a[b] / a[0]b quickly", 500 ms limit), which sometimes go over the limit while Stryker runs 16 workers in parallel. fast-check didn't kill any of the varying mutants. break: 93 leaves room for this noise, so the weekly job doesn't fail at random. Checked: with break: 99 the run exits 1 and still writes the report.

Surviving mutants

Left for a follow-up test: PR, as planned.

Survived all 4 JSON runs:

  • src/transform.ts:64: value === 'true' → false or value === "". No test checks that &a=true gives true. Real gap.
  • src/index.ts:176-177: writable/enumerable: true → false in the defineProperty for prototype names. These entries are removed before returning, so probably equivalent.
  • src/index.ts:367: [] → ["Stryker was here"]. Needs a closer look.
  • src/index.ts:377: pathPart.type === 'object' → true. Equivalent: array path parts are digits or empty, never on Object.prototype.
  • src/index.ts:407: isRemoved && nextPathValue === undefined → isRemoved && true. Needs a closer look.
  • src/path-parts.ts:47: if (open > lastClose) return -1 → if (false). Needs a closer look.
  • src/path-parts.ts:119: close < idx → close <= idx. Probably equivalent.

Only ever killed by a timing test (1–2 of 4 runs): index.ts:166 (if (!isPrototypeKey(key)) → false, and its block emptied; sends every key through the slower defineProperty, so partly a real perf kill), path-parts.ts:47 (> → >=), path-parts.ts:49 (< → <=), path-parts.ts:117 (< → <=), path-parts.ts:122 (> → >=). The loop-bound ones add at most one loop step, so those kills are noise.

npm audit

npm audit reports 5 moderate findings, all in Stryker's dev-only dependency tree:

  • minimatch 10.2.6 (GHSA-g3hv-75wg-m249, glob pattern amplification; the patterns come from our config)
  • qs 6.15.1 via typed-rest-client (used by the dashboard reporter, which we don't use)

typed-rest-client pins qs exactly, so only an overrides entry could fix it. main has 0 findings. CI doesn't run npm audit, and nothing ships in the package.

npm run validate passes (knip is happy with stryker.config.mjs; I dropped a JSDoc type import from @stryker-mutator/api that knip reported as an unlisted dependency). The workflow can only be started with workflow_dispatch once it's on main.

Add Stryker with the vitest runner, `npm run mutate`, and a `mutate`
workflow (weekly and workflow_dispatch, not on PRs) that uploads the
HTML report as an artifact.

Mutate src/index.ts, src/path-parts.ts and src/transform.ts: the parser
was split into modules after the plan said "only src/index.ts".
errors.ts is left out (constructors only).

Stryker's tsconfig preprocessor calls ts.parseConfigFileTextToJson,
which TypeScript 7 doesn't ship, so every run crashed. It only rewrites
extends/references paths, which our tsconfig doesn't have; pointing
tsconfigFile at a missing file skips it.

The first five runs scored 95.7-97.0% (fast-check data differs per run),
so thresholds.break is 93.
@codecov-commenter

codecov-commenter commented Nov 9, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (13119f1) to head (fcba893).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff            @@
##              main       #18   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            4         4           
  Lines          134       134           
  Branches        45        45           
=========================================
  Hits           134       134           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The kills that vary between runs all come from the timing tests that
parse 200 000-character paths (500 ms limit), which sometimes fail
under Stryker's parallel load. fast-check didn't kill any of them.
@milamer
milamer merged commit d51be48 into main Nov 9, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants