Support running MCP conformance tests against the Ruby SDK#248
Draft
koic wants to merge 1 commit intomodelcontextprotocol:mainfrom
Draft
Support running MCP conformance tests against the Ruby SDK#248koic wants to merge 1 commit intomodelcontextprotocol:mainfrom
koic wants to merge 1 commit intomodelcontextprotocol:mainfrom
Conversation
Member
Author
|
@modelcontextprotocol/ruby-sdk I'm currently working on conformance / tiering, and this PR provides functionality required for that effort. Issues identified during the development of this PR have been addressed in the following PRs. Once these PRs are resolved, the remaining When you have a moment, could you please help move the reviews forward?
|
koic
force-pushed
the
conformance
branch
6 times, most recently
from
7d2dfc7 to
aea19b6
Compare
9 tasks
Contributor
|
This is great! |
koic
force-pushed
the
conformance
branch
3 times, most recently
from
8f5bf37 to
9df33a6
Compare
koic
added a commit
that referenced
this pull request
Mar 1, 2026
`Errno::ECONNRESET` is a `SystemCallError`, not an `IOError`, so it was not caught by the existing `rescue IOError, Errno::EPIPE` clauses. When a client disconnected with a TCP RST during SSE notification broadcast, the unhandled exception aborted the entire broadcast loop, causing notifications to be lost for all remaining sessions. This caused the `tools-call-with-logging` conformance scenario to fail in CI (#248), where stale sessions from prior scenarios triggered `ECONNRESET` on the first notification write, aborting the broadcast and losing log notifications. Add `Errno::ECONNRESET` to all four rescue clauses in `StreamableHTTPTransport`: `send_notification` (targeted and broadcast), `send_response_to_stream`, and `send_keepalive_ping`.
koic
force-pushed
the
conformance
branch
5 times, most recently
from
f821f0e to
0114ae1
Compare
## Motivation To assess and track the Ruby SDK's compliance with the MCP specification, integrate `@modelcontextprotocol/conformance` as the standard evaluation harness. This makes it possible to measure conformance objectively, catch regressions early, and produce the kind of evaluation data needed for SDK tier assessments. ## Scope Only server conformance is included at this point. Client conformance requires protocol-level features that MCP::Client does not yet support (initialize handshake, notifications, ping, etc.). ## Usage Run conformance tests: ```console bundle exec rake conformance ``` Start the conformance server in the foreground for iterating on a single scenario without the per-run startup overhead: ```console bundle exec rake conformance_server ``` ## Default Rake Task (`bundle exec rake`) It is configured to run in order of lower execution cost: `rubocop`, `test`, and then `conformance`. ## CI `.github/workflows/conformance.yml` file has been added to run the conformance tests in CI. ## SDK Tier Report The MCP SDK Tier system (https://modelcontextprotocol.io/community/sdk-tiers) requires SDK maintainers to self-assess and report results to the SDK Working Group via https://github.com/modelcontextprotocol/modelcontextprotocol/issues. To generate a full tier assessment report, use the `/mcp-sdk-tier-audit` slash command from the https://github.com/modelcontextprotocol/conformance repository with the conformance server running: ```console # Terminal 1 (this repository): start the conformance server bundle exec rake conformance_server # Terminal 2 (conformance repository): run the tier audit skill as a slash command in Claude Code /mcp-sdk-tier-audit /path/to/modelcontextprotocol/ruby-sdk http://localhost:9292/mcp ``` The skill evaluates conformance pass rate, issue label taxonomy, triage metrics, documentation coverage, and policy compliance, then produces a markdown report suitable for tier advancement submissions.
Contributor
🎉
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation and Context
To assess and track the Ruby SDK's compliance with the MCP specification, integrate
@modelcontextprotocol/conformanceas the standard evaluation harness. This makes it possible to measure conformance objectively, catch regressions early, and produce the kind of evaluation data needed for SDK tier assessments.Scope
Only server conformance is included at this point. Client conformance requires protocol-level features that MCP::Client does not yet support (initialize handshake, notifications, ping, etc.).
Usage
Run conformance tests:
bundle exec rake conformanceStart the conformance server in the foreground for iterating on a single scenario without the per-run startup overhead:
bundle exec rake conformance_serverHow Has This Been Tested?
Default Rake Task (
bundle exec rake)It is configured to run in order of lower execution cost:
rubocop,test, and thenconformance.And
.github/workflows/conformance.ymlfile has been added to run the conformance tests in CI.Breaking Changes
None.
Types of changes
Checklist
Additional context
The MCP SDK Tier system (https://modelcontextprotocol.io/community/sdk-tiers) requires SDK maintainers to self-assess and report results to the SDK Working Group via https://github.com/modelcontextprotocol/modelcontextprotocol/issues.
To generate a full tier assessment report, use the
/mcp-sdk-tier-auditslash command from the https://github.com/modelcontextprotocol/conformance repository with the conformance server running:The skill evaluates conformance pass rate, issue label taxonomy, triage metrics, documentation coverage, and policy compliance, then produces a markdown report suitable for tier advancement submissions.