Adding MCP problem type basic sampler infrastructure by guidorc · Pull Request #1628 · WebFuzzing/EvoMaster

guidorc · 2026-07-04T16:18:59Z

Summary

Adding the basic search infrastructure for blackbox fuzzing of Model Context Protocol (MCP) servers. The approach is: connect to the target server, discover its capabilities via the standard MCP handshake, automatically construct a mutable input representation (gene tree) for each tool and resource from their JSON Schema definitions, then use a search algorithm to evolve a sequences of calls that maximise coverage; reaching each capability and obtaining a non-error response

Components

MCP client (HttpMcpClient, McpClient, McpDTOs)

An HTTP-based client that implements the MCP JSON-RPC protocol: initialization handshake, tool invocation (tools/call), static resource reads (resources/read), and capability discovery (tools/list, resources/list, resources/templates/list).

Action model (McpAction, McpToolCallAction, McpResourceReadAction)

Two concrete action types mirror the two MCP interaction patterns: calling a named tool with structured arguments, and reading a resource by URI (including template URIs with mutable parameters).

Individual (McpIndividual)

A test case represented as an ordered sequence of McpActions, integrating with EvoMaster's existing EnterpriseIndividual and mutation infrastructure.

Sampler (McpSampler)

On startup, connects to the target MCP server, performs the handshake, and discovers all tools and resources. Each tool's JSON Schema input definition is converted into an ObjectGene tree used by the search engine to mutate arguments. Pre-built single-action individuals ensure every capability is exercised at least once before random search begins.

Fitness function (McpBlackBoxFitness)

Evaluates individuals by executing their actions sequentially and scoring coverage targets based on observable protocol signals: 1.0 for a successful tool call or resource read, 0.5 for a server-reported error (partial credit keeps the search gradient useful). Exceptions break the action sequence early.

McpBlackBoxModule

Wires all the above into EvoMaster's dependency injection container and registers the problem type in the main entry point.

Follow Up

McpTestCaseWriter — the test output pipeline currently uses NoTestCaseWriter, so generated test files have empty bodies. A dedicated writer that emits executable MCP
client calls is needed.
Format-aware gene building — string fields carrying semantic formats (date-time, email, uuid) are built as plain StringGene. The sampler needs a three-layer cascade
(explicit JSON Schema format → type switch → name-based heuristics) to generate valid values for these fields, matching the approach used by the REST builder.
SortingHelper / TestCaseNamingStrategyFactory — McpIndividual is not yet registered in the test sorting and naming strategies, causing a handled warning during output.

@PostConstruct

Introduces a new problem domain for blackbox testing of MCP (Model Context Protocol) servers, following the same architecture as the existing GraphQL and RPC problem domains. New components under core/src/main/kotlin/.../problem/mcp/: - McpAction (abstract base), McpToolCallAction (tools/call), McpResourceReadAction (resources/read), McpInputParam, McpUriParam - McpIndividual: chromosome wrapping sequences of MCP actions - McpCallResult: per-action execution record storing isError flag - client/McpClient (interface), client/HttpMcpClient (JSON-RPC 2.0 over HTTP with pagination), client/McpDTOs - service/McpSampler: @PostConstruct discovery via tools/list, resources/list, resources/templates/list; gene tree building from JSON Schema; random and smart (ad-hoc) sampling - service/McpFitness (abstract base), service/McpBlackBoxFitness: per-action scoring using isError flag and successful reads as signals - service/McpBlackBoxModule: Guice bindings mirroring GraphQLBlackBoxModule Unit tests cover action construction, individual copy/mutation, and HttpMcpClient JSON-RPC parsing via WireMock stubs.

- Add ProblemType.MCP (experimental) to EMConfig enum - Add bbTargetUrl validation for MCP black-box mode - Route ProblemType.MCP to McpBlackBoxModule in Main.init() - Add getAlgorithmKeyMcp() supporting RANDOM, MIO, MOSA, WTS, SMARTS - Dispatch MCP algorithm key in Main.run() - Bind NoTestCaseWriter in McpBlackBoxModule EvoMaster can now be invoked with: --blackBox true --problemType MCP --bbTargetUrl <url>

@PostConstruct

Connection failures during @PostConstruct were surfacing as an uncaught InvocationTargetException labelled as an EvoMaster bug. Wrap the listTools() call so a failed connection to the MCP server produces a clean, user-readable SutProblemException instead.

Servers may return 400/404/405 for MCP methods they don't support (e.g., resources/list on a tools-only server). Previously this caused an IOException from HttpURLConnection.inputStream on non-2xx responses, crashing the sampler during initialization. Extracted openConnection() helper and made post() return null on 400/404/405, which callers treat as an empty/error result.

* Setting up MCP Problem Type Skeleton * remove db related size from MCP individual * Fix for resource read action id Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * feedback * adjust resource read action parameters --------- Co-authored-by: guido-rodriguez_sfemu <guido.rodriguez@salesforce.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Pgarrett · 2026-07-12T18:04:56Z

+                "id" to nextId()
+            )
+        )
+        val conn = URL(baseUrl).openConnection() as HttpURLConnection


Any particular reason for using this connection method and not something like the Jersey client? Like GraphQL or REST do so

No particular reason, just unfamiliar with the Jersey client. We should be able to use it here as well 👍

Pgarrett · 2026-07-12T18:08:10Z

+        )
+        val conn = URL(baseUrl).openConnection() as HttpURLConnection
+        conn.requestMethod = "POST"
+        conn.setRequestProperty("Content-Type", "application/json")


If we can use the jersey http client, we should be using the constants provided by the library, in order to avoid these strings

Pgarrett · 2026-07-12T18:11:52Z

+        val (conn, _) = openConnection(method, params)
+        val status = conn.responseCode
+        if (status == 400 || status == 404 || status == 405) {
+            return null


Under which circumstances would these status code mean anything valid? In other words, is null a valid return here? As an empty map or a map with content are? If not, then we should throw an exception with the corresponding error. Maybe there's an error on the MCP, or at least log a warning for the user

Left over from debugging with invalid MCP server, removed the catch since we should not silently drop these cases

Pgarrett · 2026-07-12T18:14:29Z

+            items.filterIsInstance<Map<String, Any?>>().forEach { t ->
+                tools.add(
+                    McpToolDefinition(
+                        name = t["name"] as? String ?: "",


Is a tool with no name a valid state?

No, will remove the fallback values for required properties

Pgarrett · 2026-07-12T18:18:10Z

+        val (conn, _) = openConnection(
+            "initialize",
+            mapOf(
+                "protocolVersion" to "2024-11-05",


Extract to constants. Is this the latest protocol version? Or the first one using streamable HTTP and as such we support as from there? Add a comment mentioning why that protocol version

Will extract into shared MCP constants file. The version was a mistake due to testing with a sample MCP server, updated it to the latest protocol version.

Pgarrett · 2026-07-12T18:50:38Z


    override fun initSeededTests(infoDto: SutInfoDto?) {
-        throw UnsupportedOperationException("MCP server analysis is not yet supported")
+        // Not supported in Phase 3


Why remove the exception if it's still not supported?

removed it for testing, will add it back

Pgarrett · 2026-07-12T18:51:23Z

+            adHocInitialIndividuals.add(ind)
+        }
+
+        for ((_, action) in resourceActionCluster) {


Avoid duplicate code

Pgarrett · 2026-07-12T18:51:45Z

+            "boolean" -> BooleanGene(name)
+            "object" -> buildObjectGeneFromSchema(name, schema)
+            "array" -> ArrayGene(name, StringGene("element"))
+            else -> StringGene(name) // fallback for unknown/null types


Rather throw an exception I'd say

Pgarrett · 2026-07-12T18:52:10Z


 import org.evomaster.core.problem.api.ApiWsIndividual
 import org.evomaster.core.problem.enterprise.EnterpriseChildTypeVerifier
+import org.evomaster.core.problem.enterprise.EnterpriseIndividual.Companion.getEnterpriseTopGroups


What's this import for? Seems unused

Left over from testing, removed it

Pgarrett · 2026-07-12T18:54:43Z

+    }
+
+    @Test
+    fun `listTools parses tool definitions correctly`() {


Not very fond of this type of test naming, I can see them only in one class

guido-rodriguez_sfemu and others added 15 commits May 28, 2026 22:11

Add missing header

badf32d

Add MCP initialization handshake

cf1c967

Merge branch 'master' into feature/mcp-blackbox-sampler

af0cdd2

merge skeleton setup

999c3e0

Fix for MCP uriTemplate

5aea547

store MCP-Session-Id on inititalize

38ef921

Merge branch 'master' into feature/mcp-blackbox-sampler

b4a4384

Adding JavaDocs

2b0e604

remove supress unchecked cast annotations

11df7b1

Merge branch 'master' into feature/mcp-blackbox-sampler

ef80412

guidorc changed the base branch from master to external-pr-mcp July 4, 2026 16:37

Merge branch 'external-pr-mcp' into feature/mcp-blackbox-sampler

e2820c6

guidorc changed the base branch from external-pr-mcp to master July 4, 2026 16:41

Merge branch 'master' into feature/mcp-blackbox-sampler

5cf695e

guidorc requested a review from Pgarrett July 7, 2026 00:57

Merge branch 'master' into feature/mcp-blackbox-sampler

e210728

Pgarrett requested changes Jul 12, 2026

View reviewed changes

guidorc changed the title ~~Adding MCP problem type basic fuzzing infrastructure~~ Adding MCP problem type basic sampler infrastructure Jul 13, 2026

guido-rodriguez_sfemu and others added 4 commits July 12, 2026 23:02

Adopt Jersey client

d240e23

Extract constants into shared file

c501ad1

use JsonNode data type for tools input schema

ed9906d

use MediaType constants

d4c0c34

guidorc force-pushed the feature/mcp-blackbox-sampler branch from c7cc742 to d4c0c34 Compare July 13, 2026 03:05

guidorc added 2 commits July 13, 2026 20:13

remove redundant HTTP status check

4e6b786

refactor MCP list response navigation

14b9b8d

guidorc added 2 commits July 13, 2026 21:50

Validate tool result includes type

76948f5

general feedback

9517441

guidorc requested a review from Pgarrett July 14, 2026 02:36

Uh oh!

Conversation

guidorc commented Jul 4, 2026

Summary

Components

MCP client (HttpMcpClient, McpClient, McpDTOs)

Action model (McpAction, McpToolCallAction, McpResourceReadAction)

Individual (McpIndividual)

Sampler (McpSampler)

Fitness function (McpBlackBoxFitness)

McpBlackBoxModule

Follow Up

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants