ci(conformance): pin harness to 0.2.0-alpha.3 with expected-failures baseline

[maxisbey]

Modernizes the conformance CI to match the typescript-sdk pattern so the job actually gates and the failure set burns down per SEP. Supersedes #1921 (the composite-action approach) — pinning the npm package directly is what typescript-sdk settled on.

Motivation and Context

The conformance jobs were pinned to 0.1.10/0.1.13 (drifted between server and client), ran with continue-on-error: true so they never gated, and had no baseline file — failures were silently ignored. The conformance harness has since gained --expected-failures (#113) and the 0.2.0-alpha line with version-aware scenario selection.

Changes

• Pin the harness via a single workflow-level CONFORMANCE_VERSION: "0.2.0-alpha.3" env var (one place to bump).
• Add .github/actions/conformance/expected-failures.yml — the path the conformance repo's known-sdks.ts already expects. Client baseline is 16 scenarios grouped by SEP (same set as typescript-sdk's baseline); server active suite is fully green so server: [].
• Drop continue-on-error: true from both jobs. The runner now exits 0 only when failures match the baseline, 1 on regressions or stale entries.
• Harden run-server.sh with a port-conflict guard, dead-process check in the readiness loop, and curl --max-time, mirroring typescript-sdk's wrapper.

The server --suite draft step (2026-07-28 scenarios) is a follow-up PR; the baseline file already has a server: placeholder for it.

How Has This Been Tested?

Run locally against the conformance harness at 0.2.0-alpha.3:

• server --suite active --expected-failures ... → 30/30 scenarios pass, 42 assertions, exit 0.
• client --suite all --expected-failures ... → 40 scenarios: 24 pass, 16 expected-fail, 0 unexpected, 0 stale, exit 0.
• Stale-entry guard: appending a passing scenario (initialize) to the baseline → exit 1 with Stale baseline entries (now passing - remove from baseline): initialize.
• bash -n, YAML parse, and pre-commit all clean.

Breaking Changes

None for SDK users. For contributors: the conformance jobs now fail on regressions instead of silently passing. run-server.sh requires CONFORMANCE_VERSION to be set when run locally (the error message points at the workflow pin).

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update

Checklist

• I have read the MCP Documentation
• My code follows the repository's style guidelines
• New and existing tests pass locally
• I have added appropriate error handling
• I have added or updated documentation as needed

Additional context

• The conformance jobs are not branch-protection-required checks, so an npm-registry blip turns them red but does not block merges. Same posture as typescript-sdk; can revisit once it has run cleanly for a while.
• Backporting run-server.sh to v1.x will need the env: CONFORMANCE_VERSION block backported in the same change (the script now reads it via ${CONFORMANCE_VERSION:?}).
• Follow-up on the conformance repo: add a python-sdk (main) entry to known-sdks.ts alongside the existing python-sdk-v1.

_{AI Disclaimer}

[felixweinberger]

I'm actually not sure all is the correct one, counterintuitively - I believe it has 2 scenarios that aren't actually load bearing (I think something related to old client auth cases from a pre-2025-11-25 spec version)

[felixweinberger]

Land it for now though, we can fix if we end up with remaining failures that aren't covered by any SEPs

[felixweinberger]

Do we not have EMA in Python at all? Do we need to build this into Python potentially @pcarleton?

[maxisbey]

yea it does not apparently