Block breaking changes to the dev schema at pull-request time - #732
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
62679f7 to
8977c18
Compare
There was a problem hiding this comment.
Pull request overview
Adds a CI gate to prevent the development schema from becoming structurally more restrictive.
Changes:
- Adds the compatibility gate and eight integration tests.
- Runs the gate before corpus validation.
- Documents the additive schema-evolution policy.
Show a summary per file
| File | Description |
|---|---|
scripts/versioning/check-dev-schema-compat.js |
Implements base-to-HEAD comparison. |
scripts/versioning/tests/dev-schema-gate-integration.test.js |
Adds end-to-end gate coverage. |
scripts/versioning/package.json |
Exposes the gate as an npm script. |
.github/workflows/Versioning.Checks.Job.yml |
Adds the CI check. |
.github/copilot-instructions.md |
Documents compatibility requirements. |
Review details
- Files reviewed: 5/5 changed files
- Comments generated: 1
- Review effort level: Balanced
| ? "" | ||
| : ` (dev line moved ${baseVersions.devSchemaFile} -> ${headVersions.devSchemaFile})`; | ||
|
|
||
| const findings = detectBreaking(baseSchema, headSchema); |
There was a problem hiding this comment.
Bot's comment seems legit? A removed field is blocked unconditionally even though the PR raises min?
Shall we address this before checking in?
There was a problem hiding this comment.
Moving this to #738 since it introduces the x-mxc-since/x-mxc-until availability metadata- https://github.com/microsoft/mxc/pull/738/changes#r3737807652
8977c18 to
04ca0fe
Compare
This PR adds per-field schema version windows: a wire field can declare the
range of config schema versions it is valid in, and the parser rejects any use
outside that range. It is what makes shape-only support for older schema
versions real — until now a window annotation would have been documentation
that nothing honoured.
Details
* New `mxc_version_derive` proc-macro crate. `#[derive(VersionWindows)]` lifts
`#[mxc_version(since = "0.8")]` / `until` off `wire.rs` into metadata that
normal builds carry, so one declaration feeds both the parser and schema
generation (published as `x-mxc-since` / `x-mxc-until`). A derive is required
rather than `#[schemars(extend(...))]`, which sits behind the `schema-gen`
feature and is invisible to the parser.
* The mechanism fails **open** if a derived JSON name ever disagrees with what
serde accepts — the window simply never fires — so that case is guarded twice:
the macro compile-errors on every serde construct it cannot model exactly
(`flatten`, split `rename`/`rename_all`, unknown `rename_all`, data-carrying
variants), and a conformance test cross-checks all 32 wire types against the
names `schemars` independently derives.
* The gate runs immediately after deserialisation in every entry point, because
`convert_wire_config` moves fields out of the config; state-aware requests are
gated on the original document, not the experimental-masked copy.
* `version` is now **required** — it selects the legal field surface, so an
absent one would silently opt out of every window.
* New `version_incompatible` code across all five surfaces (Rust `MxcErrorCode`,
engine `ErrorCode`, TS, C#, `MXC_STATUS_VERSION_INCOMPATIBLE = 13`) carrying
`details: { field, declaredVersion, since, until }`. The supported-range error
migrates onto it. NOTE: this changes an existing error's observable shape — a
consumer string-matching the old range message is affected.
* Three annotations, each checked against real corpus usage first: `seatbelt`
since 0.7, `processContainer.captureDenials` / `learningMode` since 0.8. The
central subtlety is what is deliberately **not** annotated: schema-first-
appearance is only a lower bound on accepted surface. `experimental` was an
open block before 0.8, and state-aware requests declare 0.6 while carrying
`phase` / `sandboxId` / `correlationVector` — annotating those from schema
data would reject configs that have always worked.
* New `check-version-windows.js` oracle gate derives each field's true first
appearance from the frozen 0.6/0.7 and dev schemas and fails on disagreement.
It is fail-closed on the roots above, which also catches windows that would
leak onto the permissive `experimental` surface via a shared type.
* Corpus and callers migrated: 61 configs versioned (state-aware to 0.6.0-alpha,
matching what the SDK emits; one-shot to 0.8.0-alpha), ~200 Rust test
literals, the PowerShell lifecycle helpers (stamped centrally), and the SDK
builders, which no longer synthesise a top-level `seatbelt` marker below 0.7.
Tests
* On the squashed tip: `cargo fmt --all -- --check`,
`cargo check --workspace --all-targets`,
`cargo clippy --workspace --all-targets -- -D warnings` and
`cargo test --workspace` all clean.
* Feature-gated builds covering every flag this diff can reach, all clean:
`wxc_common` {schema-gen, microvm}, `mxc_ffi` {dotnetsdk}, `mxc_engine`
{isolation_session}, `wxc` {isolation_session, microvm, tier2_bfs, wslc,
hyperlight}.
* `wxc_common` 594 unit tests plus a new corpus test asserting all 195 configs
declare a version and still parse, with the out-of-range fixture pinned as a
negative case by exact code and bounds. Versioning gate tests 71 → 94.
* Node SDK build + 223 tests; C# SDK 35 tests; ErrorCode parity 17 codes;
bindings codegen OK. All 10 CI gates pass.
* Non-regression: the PR #676 replay still yields exactly 6 findings naming
`allowLocalNetwork`, `allowedHosts`, `blockedHosts`, `defaultPolicy`,
`enforcementMode`, `proxy`; detector baselines hold (dev vs dev = 0,
0.6→0.7 = 6, 0.7→dev = 12); #732's gate passes; `SUPPORTED_VERSION` unchanged
at `>=0.6, <=0.8`.
* Converged through a 2-round adversarial review (14 findings; 12 fixed, 1
pushback accepted, 1 pre-existing). Two blockers were genuine test failures an
earlier verification pass had masked with a faulty grep.
* **Not executed on this host** (Windows): the macOS Seatbelt paths, the Windows
Sandbox and IsolationSession PowerShell lifecycle suites, and the host-gated
MicroVM / Hyperlight E2E configs. The macOS code does **cross-compile** —
`cargo check --target aarch64-apple-darwin --all-targets` is clean for
`mxc_engine`, `wxc_common` and `mxc-sdk`, including the new
`cfg(target_os = "macos")` regression tests — but it has not been run.
(`mxc_darwin` fails that cross-check on a pre-existing `build.rs` issue
unrelated to this change: it embeds Windows version info unconditionally.
Reproduced at the base commit.)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 21bb36ae-131a-4ab6-b062-a830ba488428
Generated-with: claude-opus-5
There was a problem hiding this comment.
Review details
Suppressed comments (2)
scripts/versioning/check-dev-schema-compat.js:89
- The gate never reads or compares
versions.min, so advancing the supported floor does not make an expired field removable:detectBreakingwill still report that deletion and fail. This contradicts the remediation printed below (and the documented policy that deletion becomes legitimate afterminpasses the field's window), leaving no working retirement path. Please make the comparison version-window-aware and add an integration case whereminadvances past anx-mxc-untilfield, or stop claiming that moving the window permits deletion.
const findings = detectBreaking(baseSchema, headSchema);
.github/copilot-instructions.md:143
- This new blanket rule conflicts with the existing experimental-schema contract in
docs/versioning.md:120-125andsrc/core/wxc_common/src/wire.rs:430-434, which says the experimental surface is in flux and may iterate without a stable compatibility promise. Because the gate compares the complete schema, narrowing a known experimental field now fails CI. Either scope compatibility enforcement to the stable surface or update the versioning policy and contributor documentation to explicitly establish the stronger experimental guarantee.
- **Dev schema compatibility**: `scripts/versioning/check-dev-schema-compat.js` is a CI gate that compares the dev schema at the pull-request base against the dev schema at HEAD and **fails on any structural restriction** — a removed property, a new `required`, a narrowed `type`, a tightened bound. There is no per-field escape hatch. Because one dev schema validates configs declaring every supported version, surface a supported version can use has to stay in it. Make a breaking change **additively**: keep the old fields, add the new shape alongside them, and let the supported-version window govern which may be used. Deleting is legitimate only once `min` in `schemas/schema-version.json` rises past the surface being dropped.
- Files reviewed: 5/5 changed files
- Comments generated: 0 new
- Review effort level: Balanced
04ca0fe to
2b42c50
Compare
There was a problem hiding this comment.
Review details
Suppressed comments (1)
scripts/versioning/check-dev-schema-compat.js:99
- Raising
mincannot actually make this remediation succeed. The gate passes only the two schema documents todetectBreaking; it never reads eithermin, and the detector explicitly ignoresx-mxc-until. Therefore deleting an expired field still produces the same finding even if this change raises the supported-version window (and a follow-up PR also compares against a base schema that still contains it), so the new gate permanently prevents the documented deletion path. Please make the comparison account for fields whoseuntilis below the new minimum, or define another safe baseline-rotation mechanism, and cover that lifecycle with an integration test.
`Configs declaring an already-supported version must keep parsing. Add ` +
`surface instead of removing it, or move the supported-version window ` +
`in the same change.`,
- Files reviewed: 5/5 changed files
- Comments generated: 0 new
- Review effort level: Balanced
2b42c50 to
972ad33
Compare
972ad33 to
52f1898
Compare
There was a problem hiding this comment.
Review details
Suppressed comments (2)
scripts/versioning/check-dev-schema-compat.js:89
- The supported-version floor cannot actually authorize a deletion here: this call always compares the complete base and HEAD schemas, and neither
baseVersions.minnorheadVersions.minaffects the comparison. Raisingminin the same PR—or in an earlier PR—therefore still reports the removed property forever, contradicting the failure guidance and documented retirement path. Filter out only surface whose version window ended before the new validated floor (or otherwise implement the floor transition), and add an integration test that raisesminwhile removing retired surface.
const findings = detectBreaking(baseSchema, headSchema);
.github/copilot-instructions.md:143
- This new versioning contract is documented only in the Copilot instructions. The repository's contributor-facing versioning design is
docs/versioning.md, and the established guidance requires versioning/process changes to update that document. Add the gate's behavior, additive migration workflow, and supported-floor retirement semantics there as well so contributors who do not use Copilot can discover the required process.
- **Dev schema compatibility**: `scripts/versioning/check-dev-schema-compat.js` is a CI gate that compares the dev schema at the pull-request base against the dev schema at HEAD and **fails on any structural restriction** — a removed property, a new `required`, a narrowed `type`, a tightened bound. There is no per-field escape hatch. Because one dev schema validates configs declaring every supported version, surface a supported version can use has to stay in it. Make a breaking change **additively**: keep the old fields, add the new shape alongside them, and let the supported-version window govern which may be used. Deleting is legitimate only once `min` in `schemas/schema-version.json` rises past the surface being dropped.
- Files reviewed: 5/5 changed files
- Comments generated: 0 new
- Review effort level: Balanced
52f1898 to
394a0db
Compare
394a0db to
bafbd1f
Compare
bafbd1f to
69f3e8b
Compare
This PR adds `scripts/versioning/lib/schema-compatibility.js`, which reports the
ways a new JSON Schema can reject an instance the old one accepted. It is the
primitive the dev-schema gate enforces with.
The detector fails closed in both directions: anything it cannot model becomes a
manual-review finding rather than silence, and anything it cannot prove is a
restriction is reported as needing proof rather than asserted as breaking.
Details
* Covers roughly thirty categories of tightening: closed objects losing a
property, new `required` entries, narrowed `type`, removed `enum` values,
tightened numeric and length bounds, added `items` / `contains` /
`propertyNames`, and changed combinators.
* Normalises equivalent spellings so a generator's rendering choice never reads
as a structural change: `const` and single-valued `enum`, `{}` and `true`,
draft-04 boolean `exclusiveMinimum` / `exclusiveMaximum`, and a `oneOf` of
singleton enums against a flat `enum`. `{}` is canonicalised during
normalisation rather than only where the diff walk enters a node, so the two
spellings stay interchangeable in positions reached by a keyword comparison as
well as by a recursive descent.
* Reports restrictions only. `integer` -> `number` is a widening, since integer
instances are a subset of number, and an assertion-free `items: true` or
`propertyNames: true` rejects nothing.
* Compares `contains` by effective `minContains` and `maxContains` rather than as
written. `contains` carries an implicit `minContains: 1`, which is what makes
even `contains: true` a restriction -- it rejects the empty array. Reading the
keywords literally would miss both halves of that: dropping an explicit
`minContains: 0` while keeping `contains` restores the default and starts
rejecting arrays with no match, while adding `contains` beside
`minContains: 0` demands nothing at all. A *changed* `contains` subschema is
routed to manual review whenever the next effective maximum is finite, because
the polarity inverts there: widening the subschema lets more elements count
toward `maxContains`, so `{contains: integer, maxContains: 1}` becoming
`{contains: number, maxContains: 1}` newly rejects `[1, 1.5]`, which a
recursive descent would read as safe.
* Preserves assertion keywords sitting beside a `$ref`. Draft 2019-09 applies
them, so returning only the target would drop a real restriction such as an
added `required` or `additionalProperties: false`. Draft-07 ignores them, so
composing is the conservative reading -- it can only ask for a review that a
draft-07 document did not need, never miss a restriction. Annotations beside a
reference are dropped, since no dialect applies them as assertions.
* Reports an unresolved reference even when both sides carry the same one:
matching text says nothing about matching content when neither target was ever
inspected. A recursion marker is treated separately, because it marks a cycle
the walk already entered, so equal markers there do mean equal structure.
* Descends into unmatched `anyOf` branches only for the exact `[T, null]`
nullable idiom, where the null branches match and leave a single possible
correspondence. That is the shape the generator emits for every optional
field, and descending is what names a property removed from inside `T`.
Nothing weaker is sound: one unmatched branch a side does not prove those
branches correspond, because a branch that did match may already cover the
removed one, and `[string, const "x"]` becoming `[string, number]` is a pure
widening. Every other shape reports that containment requires manual proof --
except one that is provable in the opposite direction: if every previous
branch still matches exactly, added branches only widen, since an instance
that matched a branch before still matches it now.
* Compares `additionalItems` only alongside tuple-form `items`, where the
keyword has effect, deciding each side's effective value from that side's own
`items` form.
* Handles hostile property names. Own-property lookups are used throughout, so a
property legitimately named `constructor` or `toString` is not skipped via the
prototype chain, and normalisation accumulates into a null-prototype object,
so a schema keyword named `__proto__` stays an own property instead of
invoking the inherited setter and vanishing from the comparison.
* Reports deterministically. Findings are sorted, and properties are descended
in name order: a normalised `$ref` target is identity-shared and a shared
subschema is reported at the first path that reaches it, so insertion order
would otherwise decide whether a finding reads `$.a` or `$.b`.
* Bounds traversal. Normalisation memoises `$ref` targets; the diff walk and
structural equality memoise node-identity pairs; combinator branches are
bucketed by a fixed-size digest, also memoised on identity. Without these a
`$ref` graph that fans out expands exponentially, and equality that serialises
its operands materialises the tree a shared graph unfolds to. Depth and node
budgets catch what remains -- including deeply nested enum data and the
untraversed payload of an unrecognised keyword, which only the equality walk
ever descends -- and surface it as a finding rather than a crash. A memo entry
seeded to break a cycle is deleted again while unwinding an aborted walk,
since the caches are keyed on node identity and an unrecognised keyword's
payload is the caller's own object: a value left behind would let a later run
clear the very pair that just exhausted the budget. Both memos are scoped to a
single call for the same reason: normalised nodes are rebuilt per call, and the
values they share with the caller -- the untraversed payloads of unrecognised
keywords -- are the caller's own mutable objects.
Tests
* 51 unit tests pass, covering each detection category, the equivalent-spelling
normalisations, `$ref` siblings, unresolved and recursive references,
prototype-named properties and keywords, tuple-only `additionalItems`,
effective `contains` bounds, deterministic ordering, and the traversal
budgets. `npm test` in `scripts/versioning` runs 79 tests across the directory
and is the exact command the Versioning Checks job runs.
* Widenings are pinned as producing no finding -- `integer` -> `number`,
`items: true`, `propertyNames: true`, `{}` in nested positions, `contains`
added beside `minContains: 0`, `contains` removed, and annotations beside a
reference -- alongside their counterparts, which are pinned as still reported:
`number` -> `integer`, `contains: true`, `minContains: 0` dropped, a narrowed
`contains` subschema, and assertion keywords beside a reference.
* Fail-closed behaviour is pinned across repeated invocations: the same objects
compared three times in a row yield the budget finding every time, rather than
clearing on the second call from a stale memo entry, and mutating a payload
between calls is detected rather than answered from the previous call's memo.
* Fan-out at depth 40 completes in 2 ms and 5,000-deep nesting returns a budget
finding instead of overflowing the stack; a 5,000-level enum value and a
20,000-deep unrecognised-keyword payload do the same.
* Combinator matching measured 20 ms at 2,000 branches, 51 ms at 4,000 and 70 ms
at 8,000.
* Regressions pin the two behaviours in tension: a removed property inside a
nullable wrapper is still named, and an equal-count branch replacement yields
one manual-proof finding rather than invented positional restrictions.
* Checked against the committed schemas: every schema compares clean against
itself, 0.6.0-alpha to 0.7.0-alpha yields 5 findings, and 0.7.0-alpha to the
dev schema yields 12.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a51edbf6-88f0-47f1-83e3-931497800904
This PR adds a CI gate that compares the dev schema at the pull-request base against the dev schema at HEAD and fails when the new one rejects an instance the old one accepted. Every other breaking-change guard compares RELEASED stable schemas, and only at release time. The surface a pull request actually edits -- the dev schema -- is unguarded, so a change can delete a stable field, regenerate the schema and the SDK types, migrate the config corpus, and merge green. PR #676 did exactly that, and was reverted by hand. Details * `scripts/versioning/check-dev-schema-compat.js` resolves the base commit with the fail-closed helper, reads both dev schemas out of git, and reports every structural restriction the compatibility detector finds. * Each side is read at its own declared `devSchemaFile`. Opening a new dev line copies the outgoing one, so the documents stay the same lineage and the comparison holds across that transition. Skipping the comparison when the line moves would let a change disable the gate by editing one line of `schemas/schema-version.json`. * A missing or unparsable schema on either side fails. The gate is only useful if it cannot succeed vacuously. * There is no per-field escape hatch. The supported-version window is what allows surface to end, so until a change moves that window, a config declaring an already-supported version has to keep parsing. * Documented in `.github/copilot-instructions.md` alongside the other schema gates, including how to make a breaking change additively, since this gate is what a contributor meets when they try to remove surface. * Runs ahead of corpus validation, because a change that removes a field also migrates the corpus; validation then passes and the removal is what needs reporting. Tests * 8 end-to-end tests drive the real CLI against throwaway repositories and assert on its exit code: unchanged and additive schemas pass; a removed property, a narrowed type, a missing schema and an unparsable schema all exit 1; a compatible new dev line passes and reports the move; and an incompatible new dev line is still blocked. * Replayed against PR #676: the gate exits 1 and names all six removed `network` fields. * Run against the repository as it stands, the gate passes, as do `check-schema-versions.js` and corpus validation across 195 configs. * Full versioning suite: 71 tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: cd48fff2-bde9-487a-ab67-012e9bbc0796 Generated-with: claude-opus-5
69f3e8b to
9ab5a41
Compare
Summary
This PR adds a CI gate that compares the dev schema at the pull-request base against the dev schema at HEAD and fails when the new one rejects an instance the old one accepted.
Every other breaking-change guard compares RELEASED stable schemas, and only at release time. The surface a pull request actually edits -- the dev schema -- is unguarded, so a change can delete a stable field, regenerate the schema and the SDK types, migrate the config corpus, and merge green. PR #676 did exactly that, and was reverted by hand.
Details
scripts/versioning/check-dev-schema-compat.jsresolves the base commit with the fail-closed helper, reads both dev schemas out of git, and reports every structural restriction the compatibility detector finds.devSchemaFile. Opening a new dev line copies the outgoing one, so the documents stay the same lineage and the comparison holds across that transition. Skipping the comparison when the line moves would let a change disable the gate by editing one line ofschemas/schema-version.json..github/copilot-instructions.mdalongside the other schema gates, including how to make a breaking change additively, since this gate is what a contributor meets when they try to remove surface.Tests
networkfields.check-schema-versions.jsand corpus validation across 195 configs.Note for reviewers
The gate blocks the dev schema accepting less than it did, which is not the same as blocking breaking changes. Because one dev schema validates configs declaring every supported version, surface a supported version can use has to remain in it. A breaking change is made additively -- keep the old fields, add the new shape alongside -- and the supported-version window (a later phase) governs which may be used at which version. Deleting becomes legitimate only once
minrises past the surface being dropped.Microsoft Reviewers: Open in CodeFlow