Add the structural schema-compatibility detector

[Gudge (MGudgin)]

Stacked on #730. Base is user/gudge/versioning_phase6a_gate_libraries; review the top commit only.

Summary

This PR adds scripts/versioning/lib/schema-compatibility.js, which reports the
ways a new JSON Schema can reject an instance the old one accepted. It is the
primitive the dev-schema gate enforces with.

The detector fails closed in both directions: anything it cannot model becomes a
manual-review finding rather than silence, and anything it cannot prove is a
restriction is reported as needing proof rather than asserted as breaking.

Details

• Covers roughly thirty categories of tightening: closed objects losing a
  property, new required entries, narrowed type, removed enum values,
  tightened numeric and length bounds, added items / contains /
  propertyNames, and changed combinators.
• Normalises equivalent spellings so a generator's rendering choice never reads
  as a structural change: const and single-valued enum, {} and true,
  draft-04 boolean exclusiveMinimum / exclusiveMaximum, and a oneOf of
  singleton enums against a flat enum. {} is canonicalised during
  normalisation rather than only where the diff walk enters a node, so the two
  spellings stay interchangeable in positions reached by a keyword comparison as
  well as by a recursive descent.
• Reports restrictions only. integer -> number is a widening, since integer
  instances are a subset of number, and an assertion-free items: true or
  propertyNames: true rejects nothing.
• Compares contains by effective minContains and maxContains rather than as
  written. contains carries an implicit minContains: 1, which is what makes
  even contains: true a restriction -- it rejects the empty array. Reading the
  keywords literally would miss both halves of that: dropping an explicit
  minContains: 0 while keeping contains restores the default and starts
  rejecting arrays with no match, while adding contains beside
  minContains: 0 demands nothing at all. A changed contains subschema is
  routed to manual review whenever the next effective maximum is finite, because
  the polarity inverts there: widening the subschema lets more elements count
  toward maxContains, so {contains: integer, maxContains: 1} becoming
  {contains: number, maxContains: 1} newly rejects [1, 1.5], which a
  recursive descent would read as safe.
• Preserves assertion keywords sitting beside a $ref. Draft 2019-09 applies
  them, so returning only the target would drop a real restriction such as an
  added required or additionalProperties: false. Draft-07 ignores them, so
  composing is the conservative reading -- it can only ask for a review that a
  draft-07 document did not need, never miss a restriction. Annotations beside a
  reference are dropped, since no dialect applies them as assertions.
• Reports an unresolved reference even when both sides carry the same one:
  matching text says nothing about matching content when neither target was ever
  inspected. A recursion marker is treated separately, because it marks a cycle
  the walk already entered, so equal markers there do mean equal structure.
• Descends into unmatched anyOf branches only for the exact [T, null]
  nullable idiom, where the null branches match and leave a single possible
  correspondence. That is the shape the generator emits for every optional
  field, and descending is what names a property removed from inside T.
  Nothing weaker is sound: one unmatched branch a side does not prove those
  branches correspond, because a branch that did match may already cover the
  removed one, and [string, const "x"] becoming [string, number] is a pure
  widening. Every other shape reports that containment requires manual proof --
  except one that is provable in the opposite direction: if every previous
  branch still matches exactly, added branches only widen, since an instance
  that matched a branch before still matches it now.
• Compares additionalItems only alongside tuple-form items, where the
  keyword has effect, deciding each side's effective value from that side's own
  items form.
• Handles hostile property names. Own-property lookups are used throughout, so a
  property legitimately named constructor or toString is not skipped via the
  prototype chain, and normalisation accumulates into a null-prototype object,
  so a schema keyword named __proto__ stays an own property instead of
  invoking the inherited setter and vanishing from the comparison.
• Reports deterministically. Findings are sorted, and properties are descended
  in name order: a normalised $ref target is identity-shared and a shared
  subschema is reported at the first path that reaches it, so insertion order
  would otherwise decide whether a finding reads $.a or $.b.
• Bounds traversal. Normalisation memoises $ref targets; the diff walk and
  structural equality memoise node-identity pairs; combinator branches are
  bucketed by a fixed-size digest, also memoised on identity. Without these a
  $ref graph that fans out expands exponentially, and equality that serialises
  its operands materialises the tree a shared graph unfolds to. Depth and node
  budgets catch what remains -- including deeply nested enum data and the
  untraversed payload of an unrecognised keyword, which only the equality walk
  ever descends -- and surface it as a finding rather than a crash. A memo entry
  seeded to break a cycle is deleted again while unwinding an aborted walk,
  since the caches are keyed on node identity and an unrecognised keyword's
  payload is the caller's own object: a value left behind would let a later run
  clear the very pair that just exhausted the budget. Both memos are scoped to a
  single call for the same reason: normalised nodes are rebuilt per call, and the
  values they share with the caller -- the untraversed payloads of unrecognised
  keywords -- are the caller's own mutable objects.

Tests

• 51 unit tests pass, covering each detection category, the equivalent-spelling
  normalisations, $ref siblings, unresolved and recursive references,
  prototype-named properties and keywords, tuple-only additionalItems,
  effective contains bounds, deterministic ordering, and the traversal
  budgets. npm test in scripts/versioning runs 79 tests across the directory
  and is the exact command the Versioning Checks job runs.
• Widenings are pinned as producing no finding -- integer -> number,
  items: true, propertyNames: true, {} in nested positions, contains
  added beside minContains: 0, contains removed, and annotations beside a
  reference -- alongside their counterparts, which are pinned as still reported:
  number -> integer, contains: true, minContains: 0 dropped, a narrowed
  contains subschema, and assertion keywords beside a reference.
• Fail-closed behaviour is pinned across repeated invocations: the same objects
  compared three times in a row yield the budget finding every time, rather than
  clearing on the second call from a stale memo entry, and mutating a payload
  between calls is detected rather than answered from the previous call's memo.
• Fan-out at depth 40 completes in 2 ms and 5,000-deep nesting returns a budget
  finding instead of overflowing the stack; a 5,000-level enum value and a
  20,000-deep unrecognised-keyword payload do the same.
• Combinator matching measured 20 ms at 2,000 branches, 51 ms at 4,000 and 70 ms
  at 8,000.
• Regressions pin the two behaviours in tension: a removed property inside a
  nullable wrapper is still named, and an equal-count branch replacement yields
  one manual-proof finding rather than invented positional restrictions.
• Checked against the committed schemas: every schema compares clean against
  itself, 0.6.0-alpha to 0.7.0-alpha yields 5 findings, and 0.7.0-alpha to the
  dev schema yields 12.

Microsoft Reviewers: Open in CodeFlow

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Adds a fail-closed structural compatibility detector for generated JSON Schemas.

Changes:

• Detects schema restrictions and unresolved comparisons.
• Normalizes equivalent schema forms and bounds traversal.
• Adds compatibility, regression, and performance tests.

Show a summary per file

File	Description
scripts/versioning/lib/schema-compatibility.js	Implements schema normalization and compatibility detection.
scripts/versioning/tests/schema-compatibility.test.js	Tests detection, normalization, references, and traversal limits.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 7
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (2)

scripts/versioning/lib/schema-compatibility.js:762

• Normalized $ref targets are identity-shared, and firstVisit suppresses every path after the first. Iterating properties in insertion order therefore decides which path receives a finding: reordering equivalent keys such as a and b that reference the same changed definition changes the report from $.a to $.b, despite the deterministic-output contract. Sort the entries by property name before descending.

  for (const [key, previousProperty] of Object.entries(previousProperties)) {

scripts/versioning/lib/schema-compatibility.js:853

• minContains has an effective default of 1 whenever contains is present. Because this loop only checks next-side keys, removing an explicit minContains: 0 while keeping contains is missed even though arrays with zero matches become invalid (for example, { contains: true, minContains: 0 } to { contains: true } currently returns no finding). Compare effective values, accounting for both the default and whether contains is active; the added-contains branch below must use the same calculation.

  for (const key of [
    "minLength",
    "minItems",
    "minContains",
    "minProperties",
  ]) {
    compareMinimum(path, key, previous, next, breaks);
  }

• Files reviewed: 2/2 changed files
• Comments generated: 2
• Review effort level: Balanced

[Gudge (MGudgin)]

Both suppressed comments from the latest review were real, and both are fixed in the amended commit. Recording them here since they have no reply thread.

schema-compatibility.js:762 -- path determinism under key reordering. Confirmed: two properties referencing the same changed definition reported $.a or $.b purely by insertion order, which contradicts the deterministic-output contract stated a few lines from the sort. Properties are now descended in name order in both directions. A test asserts the report is identical for ["a","b"] and ["b","a"] orderings of the same document.

schema-compatibility.js:853 -- effective minContains / maxContains. Confirmed, and this was the more serious of the two: a missed restriction rather than an invented one. {contains: true, minContains: 0} -> {contains: true} returned no finding, although the implicit minContains: 1 starts rejecting arrays with no matching element.

contains is now compared by effective value rather than as written, and the added-contains branch uses the same calculation, as suggested. That fixes the reverse case too, which was a false positive in the other direction: adding contains beside minContains: 0 demands nothing, and previously reported two findings (including a nonsensical minContains increased to 0). Both keywords are also treated as inert without contains.

Pinned by tests in both directions:

Change	Before	Now
{contains, minContains: 0} -> {contains}	(missed)	minContains increased to 1
{} -> {contains, minContains: 0}	2 findings	(none)
{} -> {contains: true}	reported	reported
minContains 1 -> 2, maxContains 5 -> 2	reported	reported
contains removed entirely	(none)	(none)
minContains changed without contains	reported	(none) -- inert

All committed-schema results are unchanged by this round: every schema still compares clean against itself, 0.6.0-alpha -> 0.7.0-alpha yields 5 findings and 0.7.0-alpha -> dev yields 12. npm test in scripts/versioning passes 76 tests.

[Copilot]

Review details

Suppressed comments (1)

scripts/versioning/lib/schema-compatibility.js:1032

• When previousRemaining is empty, every old anyOf branch was matched exactly in the new schema, so adding only new branches is provably a widening. This condition nevertheless emits a manual-review finding for {anyOf:[string]} -> {anyOf:[string, number]}, contrary to the detector's restrictions-only contract. Only require proof when at least one previously accepted branch is unmatched.

      key === "anyOf" &&
      !forcedPairing &&
      (previousRemaining.length > 0 || nextRemaining.length > 0)
    ) {

• Files reviewed: 2/2 changed files
• Comments generated: 2
• Review effort level: Balanced

[Gudge (MGudgin)]

The suppressed comment from the latest review (schema-compatibility.js:1032) was correct and is fixed in the amended commit.

{anyOf: [string]} -> {anyOf: [string, number]} demanded manual proof although it is provably a widening: anyOf accepts an instance matching any branch, so if no previously accepted branch was removed, an instance that matched a branch before still matches that same branch now. The condition now requires previousRemaining.length > 0, so proof is asked for only when a previously accepted branch is actually gone.

A removed branch still reports, an equal-count replacement still reports, and the [T, null] descent is unaffected.

---

Worth noting where these last rounds have been landing. The nine-axis adversarial review that ran against this branch on 7/31 covered commit 38bf722, when the library was 626 lines; it is 1,106 now, and it flagged the $ref sibling loss, the unbounded $ref expansion, the quadratic combinator match and the {}/true asymmetry. The memoisation caches, the firstVisit dedupe, the composition marker and the effective-contains logic were all written as the fixes for those findings -- and every finding since has landed in that new code, including two I introduced myself in the previous round. The remaining lesson is that a fix round needs its own review pass rather than only a re-run of the case that prompted it.

All committed-schema results are unchanged across these rounds: every schema compares clean against itself, 0.6.0-alpha -> 0.7.0-alpha yields 5 findings and 0.7.0-alpha -> dev yields 12. npm test in scripts/versioning passes 79 tests.

[Copilot]

Review details

Suppressed comments (3)

scripts/versioning/lib/schema-compatibility.js:164

• The node budget does not cover this traversal: unrecognized keyword payloads are copied through normalization, while sameValue checks only depth. A wide, shallow extension object/array can therefore traverse arbitrarily many nodes; under a combinator, structuralDigest walks the same payload without a node count too. Carry a per-comparison node counter through these walkers so oversized input returns the promised budget finding instead of consuming unbounded CI time and memory.

  // Unrecognised keywords' values are copied through normalisation untraversed,
  // so their nesting is not covered by the normalisation budget and only this
  // walk ever descends them. Bound it here, or a deep enough extension value
  // exhausts the JS stack instead of producing the promised finding.
  if (depth > MAX_DEPTH) throw new BudgetExceeded("depth");

scripts/versioning/lib/schema-compatibility.js:8

• $schema is not inert metadata: changing it changes the dialect that interprets the document. Because it is dropped here and $ref siblings are composed on both sides, changing a schema containing $ref plus required from draft-07 to 2019-09 normalizes identically and returns no finding, although the sibling starts applying and can reject previously valid instances. Keep $schema in the comparison so a dialect change at least requires manual review.

  "$schema",

scripts/versioning/lib/schema-compatibility.js:874

• additionalProperties: false does not prove this newly named property was previously forbidden: an unchanged patternProperties entry may already allow it. For example, adding properties.x: {type: "string"} to a closed object that already has patternProperties: {"^x$": true} is a tightening, but this continue skips it and the detector returns no finding. Account for matching previous patterns—by comparing their effective schema or conservatively requiring manual review—before skipping.

    if (has(previousProperties, key) || previousAdditional === false) continue;

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (2)

scripts/versioning/lib/schema-compatibility.js:8

• $schema selects the JSON Schema dialect; it is not semantically ignorable metadata. Dropping it can clear a real restriction when only the dialect changes—for example, the same $ref with a required sibling accepts {} under draft-07 but rejects it under 2019-09, while this normalizer composes the sibling on both sides and returns no finding. At minimum, report a changed dialect for manual review, or make normalization dialect-aware.

  "$schema",

scripts/versioning/lib/schema-compatibility.js:1013

• splice shifts the remainder of the bucket on every match, so a combinator containing many identical branches remains quadratic despite digest bucketing. The 200,000-node budget still permits tens of thousands of duplicate branches, enough for billions of element moves before the traversal guard helps. Since bucket order is irrelevant, remove a match by swapping in the last entry and popping it.

      if (index >= 0) bucket.splice(index, 1);

• Files reviewed: 2/2 changed files
• Comments generated: 2
• Review effort level: Balanced

[Soham Das (SohamDas2021)]

The 4 inline findings below are all fail-opens.

[Copilot]

Review details

Suppressed comments (3)

scripts/versioning/lib/schema-compatibility.js:869

• Removing an explicit property from a closed object is not necessarily restrictive when an unchanged patternProperties entry still accepts that name. For example, removing properties.x: {type: "string"} while retaining patternProperties: {"^x$": {type: "string"}} and additionalProperties: false leaves the accepted set unchanged, but this branch reports a definite removal. Account for matching patterns, or route this case to manual proof rather than asserting a restriction.

    } else if (nextAdditional === false) {
      breaks.push(`${path}.${key}: property removed from a closed object`);
    } else if (nextAdditional && typeof nextAdditional === "object") {
      diffNode(`${path}.${key}`, previousProperty, nextAdditional, breaks);

scripts/versioning/lib/schema-compatibility.js:627

• properties: {} is an assertion-free schema, but this predicate does not recognize it. As a result, detectBreaking({}, { properties: {} }) reports that an unconstrained schema became constrained even though both schemas accept every instance; the same applies to other valid no-op forms such as required: [] and patternProperties: {}. Normalize these no-op keywords away (or include them in the unconstrained check) so the detector preserves its restriction-only contract.

This issue also appears on line 866 of the same file.

  return Object.entries(node).every(
    ([key, value]) =>
      ANNOTATIONS.has(key) || (key === "additionalProperties" && value === true)
  );

scripts/versioning/lib/schema-compatibility.js:124

• The pre-hash encoding is ambiguous because schema keys may contain both separator characters. For example, branches {type:"string", a:1, b:2} and {type:"string", ["a\u0000p:1\u0001b"]:2} produce the same joined parts before SHA-1. Exact equality prevents a wrong match, but many such branches collapse into one bucket and make the findIndex matching quadratic again, defeating the traversal-hardening goal. Length-prefix or otherwise frame each key and child digest before hashing.

      : Object.keys(node)
          .sort()
          .map((key) => `${key}\u0000${structuralDigest(node[key], depth + 1)}`);

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (3)

scripts/versioning/lib/schema-compatibility.js:164

• This equality walk enforces only the depth limit; it never charges visited pairs against MAX_NODES. A shallow unrecognized-keyword payload with hundreds of thousands of children therefore bypasses the advertised traversal budget and is fully walked, allocating large key arrays and potentially stalling or exhausting CI. Add a call-local node counter to this walk and raise BudgetExceeded("size") when it reaches the same bound.

  // Unrecognised keywords' values are copied through normalisation untraversed,
  // so their nesting is not covered by the normalisation budget and only this
  // walk ever descends them. Bound it here, or a deep enough extension value
  // exhausts the JS stack instead of producing the promised finding.
  if (depth > MAX_DEPTH) throw new BudgetExceeded("depth");

scripts/versioning/lib/schema-compatibility.js:1053

• The nullable special case is not actually proven disjoint: some(isNullBranch) also accepts [null, null]. Changing {anyOf:[{type:"null"},{type:"null"}]} to {anyOf:[{type:"null"},{type:"string"}]} is a pure widening, but the unmatched null/string branches are force-paired and reported as removing null, even though the matched null branch still accepts it. Only force-pair when there is exactly one null branch and the other branch is proven not to accept null; otherwise use the manual-proof finding.

    const nullableIdiom =
      previousBranches.length === 2 &&
      nextBranches.length === 2 &&
      previousBranches.some(isNullBranch) &&
      nextBranches.some(isNullBranch);

scripts/versioning/lib/schema-compatibility.js:82

• Enum data is guarded only by depth, not by MAX_NODES. A very wide, shallow enum object/array is fully copied and stringified, and sorting recomputes that canonical form repeatedly, so this path can consume unbounded time and memory despite the detector's traversal-budget contract. Charge enum-data nodes to a call-local size budget (and preferably memoize each value's canonical form during sorting).

function canonicalValue(value, depth) {
  if (depth > MAX_DEPTH) throw new BudgetExceeded("depth");

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (1)

scripts/versioning/lib/schema-compatibility.js:354

• This normalization excludes the generator's actual singleton-enum shape because each generated branch also carries type: "string" (for example, schemas/dev/mxc-config.schema.0.8.0-dev.json:74-89). Therefore a generated oneOf compared with the equivalent flat { type: "string", enum: [...] } still produces a manual-review finding, contrary to the stated equivalent-spelling normalization. Accept and preserve a common redundant branch type, and add a oneOf-to-flat-enum regression test; the current test only compares oneOf with oneOf.

    node.oneOf.every(
      (branch) =>
        branch &&
        Array.isArray(branch.enum) &&
        branch.enum.length === 1 &&
        Object.keys(branch).every(
          (key) => key === "enum" || ANNOTATIONS.has(key)
        )

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (1)

scripts/versioning/lib/schema-compatibility.js:168

• This bounds only depth, not the number of values visited. Because unrecognized-keyword payloads are copied through normalization without incrementing ctx.nodes, a shallow payload such as a very large flat array bypasses MAX_NODES; equal payloads make sameValue traverse all entries, and the same gap exists when structuralDigest hashes such a payload inside a combinator. Thus a committed schema can still consume unbounded comparison time/memory instead of returning the promised budget finding. Track a per-call node count in these auxiliary walks as well.

  if (depth > MAX_DEPTH) throw new BudgetExceeded("depth");

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (2)

scripts/versioning/lib/schema-compatibility.js:1061

• some(isNullBranch) does not prove that this is a disjoint [T, null] wrapper. For example, [type:null, type:null] changing to [type:null, type:number] is a widening, but one null branch matches and the leftovers are force-paired, producing a false null-removed finding; T spelled as enum: [null] has the same overlap. Only force the pairing when there is exactly one null arm and the unmatched arm is provably non-null; otherwise use the manual-proof path.

      key === "oneOf" &&
      (previousRemaining.length > 0 || nextRemaining.length > 0)
    ) {
      breaks.push(
        `${path}: oneOf branches changed; exactly-one compatibility requires manual proof`
      );
      continue;
    }

scripts/versioning/lib/schema-compatibility.js:542

• These keywords have an effective default of 0, so adding minLength: 0, minItems: 0, or minProperties: 0 does not reject anything. Treating every previously absent value as an increase produces a false breaking finding (for example, { type: "string" } to { type: "string", minLength: 0 }). Compare against the effective default when the old keyword is absent.

  ) {
    breaks.push(`${path}: lower bound was tightened`);
  }

  const previousUpper = upperBound(previous);
  const nextUpper = upperBound(next);

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (2)

scripts/versioning/lib/schema-compatibility.js:168

• This bounds nesting depth but not the number of visited nodes. Because unknown-keyword payloads bypass normalization, a shallow payload with hundreds of thousands of array elements or object members is fully traversed here without ever hitting MAX_NODES; the same gap exists in the enum canonicalization and digest walks. That breaks the advertised bounded-traversal guarantee and can make the versioning gate consume unbounded CPU/memory. Thread a per-comparison node budget through these auxiliary traversals as well.

  // Unrecognised keywords' values are copied through normalisation untraversed,
  // so their nesting is not covered by the normalisation budget and only this
  // walk ever descends them. Bound it here, or a deep enough extension value
  // exhausts the JS stack instead of producing the promised finding.
  if (depth > MAX_DEPTH) throw new BudgetExceeded("depth");

scripts/versioning/lib/schema-compatibility.js:559

• The absence of minLength, minItems, and minProperties is equivalent to a value of 0. This branch therefore reports a restriction for additions such as minItems: 0, even though they reject no previously accepted instance. Compare against the implicit zero default when the previous keyword is absent.

    has(next, key) &&
    (!has(previous, key) || next[key] > previous[key])

• Files reviewed: 2/2 changed files
• Comments generated: 0 new
• Review effort level: Balanced