feat: added mcp as a test type

[constantinius]

Closes https://linear.app/getsentry/issue/TET-1871/ai-testing-framework-add-mcp-type

[linear]

TET-1871 AI Testing Framework: Add MCP type

In general, to fix a “replacement of a substring with itself” issue, either remove the redundant .replace call if no transformation is needed, or change the replacement string so it reflects the intended transformation. Here, we should avoid changing the program’s behavior beyond removing the no-op; altering the output for "sse" (e.g., to "se" or "e") would be speculative and might break expectations elsewhere.

The best minimal fix is therefore to delete the .replace("sse", "sse") call from the abbreviateMode function in .github/scripts/compare-ctrf.cjs. This keeps the existing abbreviations unchanged and simply stops doing a useless, confusing operation. Concretely, update the function body so it ends with .replace("lowlevel", "lo") as the last replacement. No new imports, methods, or additional definitions are required.

[github-actions]

🔴 AI SDK Integration Test Results

Status: 3 regressions detected

Summary

Metric	main	PR	Change
Total Tests	384	424	+40
Passed	190	224	+34 ✅
Failed	194	200	+6 ⚠️

🔴 Regressions

These tests were passing on main but are now failing:

node/langchain :: Multi-Turn LLM Test (streaming)

Error: 1 check(s) failed:

1 check(s) failed:
Input token progression failed: tokens should increase with each turn
gen_ai.response.model is missing (optional but recommended)
gen_ai.response.model is missing (optional but recommended)
gen_ai.response.model is missing (optional but recommended)

node/langchain :: Multi-Turn LLM Test (blocking)

Error: 1 check(s) failed:

1 check(s) failed:
Input token progression failed: tokens should increase with each turn
gen_ai.response.model is missing (optional but recommended)
gen_ai.response.model is missing (optional but recommended)
gen_ai.response.model is missing (optional but recommended)

node/openai :: Multi-Turn LLM Test (blocking)

Error: 1 check(s) failed:

1 check(s) failed:
Input token progression failed: tokens should increase with each turn

✅ Fixed

These tests were failing on main but are now passing:

• python/google-genai :: Conversation ID LLM Test (sync, blocking)

🆕 New Tests

Passing (36):

• ✅ node/mcp :: Basic MCP Tool Call Test (stdio)
• ✅ node/mcp :: Basic MCP Tool Call Test (sse)
• ✅ node/mcp :: MCP Tool Error Test (stdio)
• ✅ node/mcp :: MCP Tool Error Test (sse)
• ✅ node/mcp :: MCP Multiple Tool Calls Test (stdio)
• ✅ node/mcp :: MCP Multiple Tool Calls Test (sse)
• ✅ node/mcp :: MCP Resource Read Test (stdio)
• ✅ node/mcp :: MCP Resource Read Test (sse)
• ✅ node/mcp :: MCP Prompt Get Test (stdio)
• ✅ node/mcp :: MCP Prompt Get Test (sse)
• ✅ python/fastmcp :: Basic MCP Tool Call Test (async, blocking, stdio)
• ✅ python/fastmcp :: Basic MCP Tool Call Test (async, blocking, sse)
• ✅ python/fastmcp :: MCP Tool Error Test (async, blocking, stdio)
• ✅ python/fastmcp :: MCP Tool Error Test (async, blocking, sse)
• ✅ python/fastmcp :: MCP Multiple Tool Calls Test (async, blocking, stdio)
• ✅ python/fastmcp :: MCP Multiple Tool Calls Test (async, blocking, sse)
• ✅ python/fastmcp :: MCP Resource Read Test (async, blocking, stdio)
• ✅ python/fastmcp :: MCP Resource Read Test (async, blocking, sse)
• ✅ python/fastmcp :: MCP Prompt Get Test (async, blocking, stdio)
• ✅ python/fastmcp :: MCP Prompt Get Test (async, blocking, sse)
• ✅ python/mcp :: Basic MCP Tool Call Test (async, blocking, stdio, highlevel)
• ✅ python/mcp :: Basic MCP Tool Call Test (async, blocking, stdio, lowlevel)
• ✅ python/mcp :: Basic MCP Tool Call Test (async, blocking, sse, highlevel)
• ✅ python/mcp :: MCP Tool Error Test (async, blocking, stdio, highlevel)
• ✅ python/mcp :: MCP Tool Error Test (async, blocking, stdio, lowlevel)
• ✅ python/mcp :: MCP Tool Error Test (async, blocking, sse, highlevel)
• ✅ python/mcp :: MCP Tool Error Test (async, blocking, sse, lowlevel)
• ✅ python/mcp :: MCP Multiple Tool Calls Test (async, blocking, stdio, highlevel)
• ✅ python/mcp :: MCP Multiple Tool Calls Test (async, blocking, stdio, lowlevel)
• ✅ python/mcp :: MCP Multiple Tool Calls Test (async, blocking, sse, highlevel)
• ✅ python/mcp :: MCP Resource Read Test (async, blocking, stdio, highlevel)
• ✅ python/mcp :: MCP Resource Read Test (async, blocking, stdio, lowlevel)
• ✅ python/mcp :: MCP Resource Read Test (async, blocking, sse, highlevel)
• ✅ python/mcp :: MCP Prompt Get Test (async, blocking, stdio, highlevel)
• ✅ python/mcp :: MCP Prompt Get Test (async, blocking, stdio, lowlevel)
• ✅ python/mcp :: MCP Prompt Get Test (async, blocking, sse, highlevel)

Failing (4):

❌ python/mcp :: Basic MCP Tool Call Test (async, blocking, sse, lowlevel)

Error: 1 check(s) failed:

1 check(s) failed:
Expected 1 MCP span(s) but found 0

❌ python/mcp :: MCP Multiple Tool Calls Test (async, blocking, sse, lowlevel)

Error: 2 check(s) failed:

2 check(s) failed:
Expected 2 MCP span(s) but found 0
Expected 2 MCP tool span(s) but found 0

❌ python/mcp :: MCP Resource Read Test (async, blocking, sse, lowlevel)

Error: 1 check(s) failed:

1 check(s) failed:
Expected 1 MCP span(s) but found 0

❌ python/mcp :: MCP Prompt Get Test (async, blocking, sse, lowlevel)

Error: 1 check(s) failed:

1 check(s) failed:
Expected 1 MCP span(s) but found 0

Test Matrix

Agent Tests

SDK	Basic Agent Test	Conversation ID Agent Test	Long Input Agent Test	Tool Call Agent Test	Tool Error Agent Test	Vision Agent Test
nextjs/mastra	❌	❌	—	❌	❌	❌
nextjs/vercel	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str
node/manual	✅	❌	✅	✅	✅	✅
node/mastra	❌	❌	❌	❌	❌	❌
node/vercel	❌	❌	❌	❌	❌	❌
php/laravel	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str
python/langgraph	❌a ❌s	❌a ❌s	❌a ❌s	❌a ❌s	❌a ❌s	❌a ❌s
python/manual	✅a ✅s	✅a ✅s	✅a ✅s	✅a ✅s	✅a ✅s	✅a ✅s
python/openai-agents	✅	✅	❌	❌	❌	❌
python/pydantic-ai	✅	✅	❌	❌	❌	❌

Embedding Tests

SDK	Basic Embeddings Test
browser/google-genai	❌
browser/langchain	❌
browser/openai	✅
nextjs/google-genai	❌
nextjs/langchain	❌
nextjs/openai	✅
nextjs/vercel	❌
node/google-genai	❌
node/langchain	❌
node/openai	✅
node/vercel	❌
php/laravel	❌
python/google-genai	❌a, blk ❌s, blk
python/langchain	❌a, blk ❌s, blk
python/litellm	❌a, blk ✅s, blk
python/manual	✅a, blk ✅s, blk
python/openai	✅a, blk ✅s, blk

LLM Tests

SDK	Basic Error LLM Test	Basic LLM Test	Conversation ID LLM Test	Long Input LLM Test	Multi-Turn LLM Test	Vision LLM Test
browser/anthropic	✅blk ❌str	✅blk ✅str	❌blk ❌str	✅blk ✅str	✅blk ✅str	✅blk ✅str
browser/google-genai	✅blk ✅str	✅blk ✅str	❌blk ❌str	❌blk ❌str	❌blk ❌str	✅blk ✅str
browser/langchain	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str
browser/openai	✅blk ✅str	✅blk ❌str	❌blk ❌str	✅blk ❌str	✅blk ❌str	✅blk ❌str
nextjs/anthropic	✅blk ✅str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str
nextjs/google-genai	✅blk ✅str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str
nextjs/langchain	✅blk ✅str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str	❌blk ❌str
nextjs/openai	✅blk ✅str	✅blk ❌str	✅blk ❌str	✅blk ❌str	✅blk ❌str	✅blk ❌str
node/anthropic	✅blk ✅str	✅blk ✅str	✅blk ✅str	✅blk ✅str	✅blk ✅str	✅blk ✅str
node/google-genai	✅blk ✅str	✅blk ✅str	✅blk ✅str	✅blk ✅str	✅blk ✅str	✅blk ✅str
node/langchain	✅blk ✅str	✅blk ✅str	✅blk ✅str	✅blk ✅str	❌📉blk ❌📉str	✅blk ✅str
node/manual	—	✅	✅	✅	✅	✅
node/openai	✅blk ✅str	✅blk ✅str	✅blk ✅str	✅blk ✅str	❌📉blk ❌str	✅blk ✅str
python/anthropic	✅a, blk ❌a, str ✅s, blk ❌s, str	✅a, blk ❌a, str ✅s, blk ❌s, str	✅a, blk ❌a, str ✅s, blk ❌s, str	✅a, blk ❌a, str ✅s, blk ❌s, str	✅a, blk ❌a, str ✅s, blk ❌s, str	✅a, blk ❌a, str ✅s, blk ❌s, str
python/google-genai	✅a, blk ✅a, str ✅s, blk ✅s, str	✅a, blk ✅a, str ✅s, blk ✅s, str	✅a, blk ✅a, str ✅🔧s, blk ✅s, str	✅a, blk ✅a, str ✅s, blk ✅s, str	❌a, blk ❌a, str ❌s, blk ❌s, str	❌a, blk ❌a, str ❌s, blk ❌s, str
python/langchain	❌a, blk ❌a, str ❌s, blk ❌s, str	✅a, blk ✅a, str ✅s, blk ✅s, str	✅a, blk ✅a, str ✅s, blk ✅s, str	✅a, blk ✅a, str ✅s, blk ✅s, str	✅a, blk ✅a, str ✅s, blk ✅s, str	✅a, blk ✅a, str ✅s, blk ✅s, str
python/litellm	❌a, blk ❌a, str ❌s, blk ❌s, str	❌a, blk ❌a, str ✅s, blk ✅s, str	❌a, blk ❌a, str ✅s, blk ✅s, str	❌a, blk ❌a, str ✅s, blk ✅s, str	❌a, blk ❌a, str ✅s, blk ✅s, str	❌a, blk ❌a, str ✅s, blk ✅s, str
python/manual	—	✅a, blk ✅s, blk	✅a, blk ✅s, blk	✅a, blk ✅s, blk	✅a, blk ✅s, blk	✅a, blk ✅s, blk
python/openai	❌a, blk ❌a, str ❌s, blk ❌s, str	✅a, blk ❌a, str ✅s, blk ❌s, str	✅a, blk ❌a, str ✅s, blk ❌s, str	✅a, blk ❌a, str ✅s, blk ❌s, str	✅a, blk ❌a, str ✅s, blk ❌s, str	❌a, blk ❌a, str ❌s, blk ❌s, str

MCP Tests

SDK	Basic MCP Tool Call Test	MCP Multiple Tool Calls Test	MCP Prompt Get Test	MCP Resource Read Test	MCP Tool Error Test
node/mcp	✅🆕sse ✅🆕io	✅🆕sse ✅🆕io	✅🆕sse ✅🆕io	✅🆕sse ✅🆕io	✅🆕sse ✅🆕io
python/fastmcp	✅🆕a, blk, sse ✅🆕a, blk, io	✅🆕a, blk, sse ✅🆕a, blk, io	✅🆕a, blk, sse ✅🆕a, blk, io	✅🆕a, blk, sse ✅🆕a, blk, io	✅🆕a, blk, sse ✅🆕a, blk, io
python/mcp	✅🆕a, blk, sse, hi ❌🆕a, blk, sse, lo ✅🆕a, blk, io, hi ✅🆕a, blk, io, lo	✅🆕a, blk, sse, hi ❌🆕a, blk, sse, lo ✅🆕a, blk, io, hi ✅🆕a, blk, io, lo	✅🆕a, blk, sse, hi ❌🆕a, blk, sse, lo ✅🆕a, blk, io, hi ✅🆕a, blk, io, lo	✅🆕a, blk, sse, hi ❌🆕a, blk, sse, lo ✅🆕a, blk, io, hi ✅🆕a, blk, io, lo	✅🆕a, blk, sse, hi ✅🆕a, blk, sse, lo ✅🆕a, blk, io, hi ✅🆕a, blk, io, lo

Legend: ✅ Pass | ❌ Fail | ✅🔧 Fixed | ❌📉 Regressed | ✅🆕 New (pass) | ❌🆕 New (fail) | 🗑️ Removed | _str=streaming _blk=blocking _a=async _s=sync _io=stdio _sse=sse _hi=highlevel _lo=lowlevel

---

Generated by AI SDK Integration Tests