WebMCP "real-user measurement"

[yoavweiss]

For in-browser agents, it might be interesting to think of ways in which developers can measure how their WebMCP implementation is doing.

While theoretically they can instrument their tool calls to see how often the agent calls them, they won't be able to know if that tool call actually corresponded to what the user was trying to achieve.

A way to measure the latter (e.g. by reflecting user's thumbs up/down votes to the site) could help enable some measurement of changes on either the WebMCP implementation or the agent's.

[bwalderman]

I think this is between the web developer and the agent implementation and should happen on the backend. As you pointed out, a developer can instrument their own tool calls but won't necessarily know if it matches what the user was trying to achieve. For that, they need insight into the agent implementation that's making the calls since only it would have that information about the user's intent. You mentioned surfacing the agent's thumbs-up/down votes to frontend script. As a web developer, I may not want just any visitor to be able to call this API and get performance data about my site's implementation. There may be privacy and fingerprinting implications as well. Feedback provided to an agent is user data that they may not want their browser sharing with the site they are visiting.

An alternative is some kind of backend solution that would need to be agent specific. For example, you embed a third-party agent on your site. The third-party agent provider has a developer dashboard on their own website where you can log in and see how your site's tools performed with their agent.

[yoavweiss]

As a web developer, I may not want just any visitor to be able to call this API and get performance data about my site's implementation.

The only data a visitor could get from this API is their own thumbs up/down data.

Maybe I should clarify what I'm thinking with regards to thumbs up/down. Whenever the user clicks thumbs up/down RE any certain interaction, the browser can make that information available to script. It might also be interesting to include e.g. the last tool call the agent performed, but not include any context that the site couldn't already capture through instrumentation.

That would work for in-browser agents, and could also work for extension agents, if browsers exposed the required extension APIs. For in-page agents, I agree that the site should have a relationship with them, and hence other means of communicating that information.

Does that make sense?

[bwalderman]

The proposal as stated makes sense. My opinion is still that this should be solved between the web developer and the agent on the backend.

I can't speak to every agent out there, but at least for an in-browser agent like Copilot in Edge, something as seemingly simple as a thumbs-up/down reaction in a chat conversation is user data and it's governed by a privacy policy. Even if the information exposed to script is just for the user's own interactions, the user presumably agreed to share that information with Copilot, not necessarily with the owner of the website they happen to be browsing. I'm not a lawyer, but if a feature to expose this information to script existed, I suspect it would need to be behind a permission dialog and some sort of notice that the user would need to agree to.

There's also the potential for abuse to consider. If a web developer relies on client-side script to retrieve this information, that makes it easier for an attacker to do things like spoof user feedback and spam HTTP calls to whatever endpoint the web developer uses to collect this information. A backend arrangement between the web developer and agent implementer is more secure, and the agent has full context of the conversation and the ability to provide higher quality data to the web developer.

I understand a solution like I'm describing doesn't exist for major in-browser agents today and so having an API to get some sort of measurement on the client side could serve as a fallback until a backend solution exists. I'm still hesitant to include something like this in the WebMCP API for the reasons above.

One more thing to consider.. It might be possible for a site to structure its tools in a way that lets the agent provide feedback, and this wouldn't require any changes to the WebMCP spec. For example, a provideFeedback tool to be called if the agent gets "stuck". Or a "feedback" parameter on a store's "checkout" tool that asks the agent to rate its experience navigating via WebMCP .