Skip to content

test: kill the surviving mutants that matter - #19

Merged
milamer merged 2 commits into
mainfrom
test/mutation-survivors
Nov 10, 2026
Merged

milamer merged 2 commits into
mainfrom
test/mutation-survivors

Conversation

@milamer

@milamer milamer commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

Follow-up to #18. Changes tests and the Stryker config only; test: doesn't release.

Changes

  • New test: &foo=true → {foo: true}. No test checked the 'true' case in defaultTransform (src/transform.ts:64), so value === 'true' → false / value === "" survived.
  • New test: a kept empty container survives a removed entry. transformEntry can return {}/[] as a value (the suite already tests that, index.test.ts "removes the entry %s with the value %j"). If a removed entry (a.toString, a[].toString) then goes through it, undoing the removed entry must not delete it: {a: {}} / {a: []}. Kills isRemoved && nextPathValue === undefined → isRemoved && true and && → || (index.ts:407).
  • New test: an empty-string property on Object.prototype doesn't remove a[] entries. Only object path parts are compared with Object.prototype (as the JSDoc says); the [] part's path is ''. Same polluted-runtime setup and cleanup as the existing "a property added to Object.prototype" tests. Kills pathPart.type === 'object' → true (index.ts:377).
  • Timing tests skipped during mutation runs. The four "parses a 200 000 character path of … quickly" tests (500 ms limit) use it.skipIf(process.env['STRYKER_MUTATOR_WORKER'] !== undefined). Stryker sets that variable in its worker processes; npm test and CI still run the tests. A local declare const process gives the type, because the project has no @types/node.
  • thresholds: {high: 100, low: 95, break: 96.3} (was break: 93).

Why skip the timing tests

Measured with the instrumented code from a Stryker sandbox and no mutant active:

a[b] a[0]b
original code 41 ms 64 ms
instrumented, alone 176 ms 188 ms
instrumented, 15 runs in parallel (like Stryker's workers) 432–576 ms 427–544 ms

Stryker compiles all mutants into the source, and every expression runs through stryMutAct_…/stryCov_… calls. The scanner loop runs once per character, so these tests went over 500 ms (7 of 30 in the parallel measurement) and "killed" whichever mutant was active. In #18's runs, every kill that varied between runs came from these tests. After the skip the score is the same in every run.

Score and threshold

96.37% (292 of 303) in every run. break: 96.3, so one new surviving mutant (291/303 = 96.04%) fails the run. Checked with the earlier threshold: removing the true test dropped the score below break and the run exited 1.

No // Stryker disable comments: the directive only accepts mutator names, not single replacements, and disables every mutant of that mutator on the line. On these lines that would also turn off mutants the tests kill today (e.g. idx < path.length → idx >= path.length next to the equivalent idx <= path.length). If the score goes up, break should be raised.

The 11 equivalent mutants

Each one was checked by a separate investigation: a proof from the code plus, for the scanner mutants, a differential search comparing the original and the mutant on millions of generated paths (all with 0 differences).

Location Mutant Why it can't change the output Search
index.ts:166 if (!isPrototypeKey(key)) → if (false), and its block → {} for a key not on Object.prototype, defineProperty with all flags true equals assignment on the library's own plain objects (same attributes, same key order) —
index.ts:176 writable: true → false every write to a prototype-named key uses defineProperty on a configurable property, and the key is deleted before returning —
index.ts:177 enumerable: true → false prototype-named children are always deleted before their container's Object.keys check (backwards walk) —
index.ts:367 createdByRemoved = [] → ["Stryker was here"] the string is processed last and destructures to a truthy container ('r') with one key, so the loop continues —
path-parts.ts:47 if (open > lastClose) return -1 → if (false) the loop doesn't run and lastClose < open is returned, which the caller skips like -1; the guard is redundant 10.2M cases
path-parts.ts:47 > → >= open === lastClose is impossible ([ vs ]) 11.1M paths
path-parts.ts:49 idx < lastClose → <= at lastClose the function returns lastClose either way 23.8M calls
path-parts.ts:117 idx < path.length → <= path[path.length] is undefined, which the loop body skips 10.8M cases
path-parts.ts:119 close < idx → <= close is -1 or after idx, never equal 0.9M cases
path-parts.ts:122 close > idx → >= same: close === idx can't happen 9.6M cases

index.ts:176, index.ts:177 and index.ts:166 can be killed only with tests we didn't add:

  • 176/177: changing Object.prototype from inside transformEntry between two entries, or (177) inspecting an object passed in with a cast after the parse threw.
  • 166: an object with an inherited setter passed in with a cast.

Those tests would pin down unrealistic or undocumented behaviour.

npm run validate passes (320 tests).

Add a test for `&foo=true` -> true: no test checked the `'true'` case
in defaultTransform, so two mutants of it survived.

Skip the 200 000-character timing tests during mutation runs
(STRYKER_MUTATOR_WORKER is set). Stryker's instrumented code runs ~4x
slower on all cores at once, so they failed without a mutant and killed
random mutants, which made the score move between runs.

The score is now 95.38% in every run. The 14 surviving mutants are
equivalent. Stryker disable comments can't exclude them: a comment
disables every mutant of that mutator on the line, which would also
turn off 11 mutants the tests kill today. Set thresholds.break to 95.3
instead, so one new surviving mutant fails the run.
@codecov-commenter

codecov-commenter commented Nov 9, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (d51be48) to head (f4d6c6c).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff            @@
##              main       #19   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            4         4           
  Lines          134       134           
  Branches        45        45           
=========================================
  Hits           134       134           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- An empty object or array that a kept entry set (via transformEntry)
  must stay when a removed entry goes through it. Kills both mutants
  of `isRemoved && nextPathValue === undefined`.
- With an empty-string property on Object.prototype, `a[]` entries
  stay: only object path parts are compared with Object.prototype.
  Kills `pathPart.type === 'object'` -> `true`.

The score is 96.37% (292 of 303) in every run; the 11 survivors are
equivalent. Raise thresholds.break to 96.3.
@milamer
milamer merged commit 769d301 into main Nov 10, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants