Skip to content

ci(conformance): run server --suite draft and baseline the 2026-07-28 scenarios#2878

Merged
maxisbey merged 1 commit into
mainfrom
conformance-draft-suite
Jun 15, 2026
Merged

ci(conformance): run server --suite draft and baseline the 2026-07-28 scenarios#2878
maxisbey merged 1 commit into
mainfrom
conformance-draft-suite

Conversation

@maxisbey

Copy link
Copy Markdown
Contributor

Adds the 2026-07-28 draft-spec server suite on top of #2877, mirroring typescript-sdk's two-step server job (active then draft). This is the second half of bringing the conformance harness up to date for the v2/2026 line — #2877 was the modernization, this adds the draft coverage so 2026 SEP work has a burn-down list.

Motivation and Context

AGENTS.md requires new 2026-07-28 features to have a passing conformance test. With #2877 the server job only runs --suite active (2025-11-25 scenarios); draft-spec server scenarios were never exercised. This adds the --suite draft step and baselines the 17 scenarios that don't pass yet, so each SEP implementation PR can remove its entries from the baseline as it lands.

Changes

How Has This Been Tested?

Run locally against @modelcontextprotocol/conformance@0.2.0-alpha.3:

  • server --suite draft --expected-failures ... → 19 scenarios, 17 expected-fail, 0 unexpected, 0 stale, exit 0. Run twice on a fresh server each time; output byte-identical.
  • server --suite active --expected-failures ... with the 17 server entries present → 30/30 pass, exit 0 (the evaluator ignores baseline entries not in the current run, so the draft entries don't trip the active step).
  • client --suite all --expected-failures ... → 24 pass + 16 expected-fail, exit 0.
  • pre-commit clean.

Breaking Changes

None.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Checklist

  • I have read the MCP Documentation
  • My code follows the repository's style guidelines
  • New and existing tests pass locally
  • I have added appropriate error handling
  • I have added or updated documentation as needed

Additional context

  • Dominant draft failure mode is -32600 Missing session IDmcp-everything-server has no stateless mode yet, so most scenarios hit the transport wall before SEP-specific checks run. Same as typescript-sdk's baseline; the entries reshuffle once SEP-2575 lands.
  • The harness's stale-entry check only fires for scenarios that actually ran, so a renamed/typo'd baseline entry would linger silently. Same gap exists for the client: block and in typescript-sdk; would need an upstream change to validate baseline names against the scenario registry.
  • run-server.sh is invoked twice now; cleanup relies on uv run forwarding SIGTERM to uvicorn. Verified to work on the sequential-restart path locally.
  • Deferred to SEP-2575 burn-down: client.py reading MCP_CONFORMANCE_PROTOCOL_VERSION (typescript-sdk doesn't either), mcp-everything-server --stateless.
  • Follow-up on the conformance repo: add a python-sdk (main) entry to known-sdks.ts.

AI Disclaimer

… scenarios

Adds the draft-spec server step on top of #2877, mirroring
typescript-sdk's two-step server job (active then draft).

- Second run-server.sh invocation with --suite draft.
- expected-failures.yml server: populated with the 17 draft scenarios
  that fail or warn (14 FAILURE + 3 WARNING-only), grouped by SEP.
  Same 17-entry set as typescript-sdk's server baseline.
- Two of the 19 draft scenarios are intentionally not baselined: their
  negative-case assertions are vacuously satisfied today by the
  stateful server's -32600 response. Noted in a trailing comment.

client: section unchanged; --suite all already covers draft client
scenarios via the PR #2877 baseline.
@maxisbey maxisbey marked this pull request as ready for review June 15, 2026 15:29
@maxisbey maxisbey merged commit 255650d into main Jun 15, 2026
31 checks passed
@maxisbey maxisbey deleted the conformance-draft-suite branch June 15, 2026 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants