feat(mcp): adopt FastMCP 4 beta and validate hosted MCP Apps

[phernandez]

Summary

Port Basic Memory Core to the current FastMCP 4 beta, validate the modern MCP protocol behavior, and use the port to fix the hosted MCP App callback contract exercised by basic-memory-cloud#1493.

The first implementation should be a narrow compatibility upgrade. Interactive input, session-backed working context, background tasks, and client response caching are valuable follow-ons, but combining them with the dependency port would make regressions hard to isolate.

Why now

FastMCP 4 moves identity into modern request metadata, makes session state explicitly request/session scoped, adds interactive input and task-backed work, and supports client-side response caching for catalog methods.

At the same time, the hosted MCP App currently runs into host-side blocks when the Vue UI calls back to a private search tool. Inspection of the Cloud app and its tenant ASGI gateway indicates the routing shim is not the failing boundary: a strict MCP Apps host can reject the callback name before FastMCP dispatches it.

Related operational evidence: #1175.

Findings

Core compatibility surface

A local FastMCP 4 beta probe found these concrete port requirements:

• Pin the exact beta and its companion packages while the release is pre-stable.
• Replace the removed fastmcp.exceptions.ToolError import path.
• Stop relying only on legacy camelCase clientInfo; support modern request metadata while preserving the legacy compatibility path during the port.
• Audit code that assumes Context state persists across independent tool calls. A probe returned 1, 1 under the modern lifecycle where the legacy assumption returned 1, 2.
• Update test fixtures that import FastMCP from the legacy MCP server module.
• Keep ChatGPT search / fetch identity behavior covered explicitly.

With a temporary exception shim, all 21 Core tools registered. A focused beta run produced 14 passes and 10 failures: nine around ChatGPT search/fetch identity and one around the client-info shape. This supports a small compatibility PR before adopting new runtime features.

Hosted Cloud path

The current Cloud data path is:

MCP App iframe
  -> app.callServerTool(...)
  -> authenticated Cloud /mcp FastMCP server
  -> private callback tool
  -> Core typed client
  -> tenant_gateway_client_factory
  -> in-process /bmc ASGI gateway
  -> TenantGateway / TenantDispatchService
  -> tenant-scoped Core FastAPI app

The factory preserves the verified access token and workspace, and the gateway re-applies tenant, user, workspace, entitlement, and project context before dispatch. Nothing in that path explains a host refusing the search callback.

The likely blocker is the callback address:

• The Cloud Vue app manually calls deterministic hashed names obtained from FastMCP's internal fastmcp.server.providers.addressing.hashed_backend_name.
• FastMCP 3.4.x hides app-only tools from tools/list; the existing Cloud test currently asserts that those callback tools are absent.
• The MCP Apps specification requires app-only tools to remain in the catalog with _meta.ui.visibility = ["app"]. The host hides them from the model but retains them as valid UI-callable tools.
• A strict host may reject an unadvertised hashed name before the request reaches the Cloud ASGI gateway.
• A FastMCP 4.0.0b1 probe advertised the plain app-only tool name with the required UI metadata. Calling the plain name succeeded. Calling the hash reached the server in the in-memory client, but emitted a warning that the tool was not listed and therefore could not be validated.

For the custom Vue app, use stable explicit callback tool names registered with app-only visibility and call those exact advertised names. Do not build the UI contract on FastMCP's private addressing helper. Prefab/function-reference apps may still use the framework's supported address wiring.

Cloud also has direct on_initialize / message.params.clientInfo assumptions in trial and product telemetry middleware; those need a modern-protocol audit when Cloud consumes the upgraded Core SHA.

Proposed delivery

Phase 1: Core compatibility port

• Pin the exact FastMCP 4 beta and compatible direct MCP/FastMCP companion constraints.
• Refresh uv.lock.
• Migrate removed imports and legacy fixtures.
• Resolve client identity from modern request metadata with a tested legacy fallback.
• Remove cross-request Context state assumptions.
• Preserve all existing public tool schemas and Core behavior.
• Add focused modern/legacy identity and lifecycle tests.
• Validate with targeted MCP tests, just fast-check, just test-smoke, and the appropriate SQLite/Postgres gates.

Phase 2: hosted MCP Apps acceptance (Cloud follow-up)

• Update Cloud's Core git pin and FastMCP dependency.
• Register custom UI callbacks as catalog-visible, app-only tools with stable explicit names.
• Make app.callServerTool use those exact advertised names.
• Replace tests that expect callbacks to be absent with assertions for app-only visibility and host/model filtering.
• Audit trial/product telemetry middleware for modern client identity and initialization lifecycle.
• Add an integration test that proves search preserves access token, workspace, tenant, and project through the ASGI gateway.
• Run a real-host acceptance test in ChatGPT; an in-memory FastMCP client does not model strict host catalog enforcement.

Phase 3: adopt new features independently

• Session state: keep a per-session current project/workspace selection, bounded by stable IDs and revalidated on each request. Do not treat Context state as cross-call storage.
• Interactive tools: prompt for project/workspace selection when intent is ambiguous instead of guessing or returning a large error.
• Background work: expose plan/apply workflows for long operations such as skill-driven bulk note reorganization, with progress, cancellation, durable receipts, and an explicit review boundary.
• Response caching: enable FastMCP client catalog caching where appropriate. This is complementary to #1172, which caches semantic/read results rather than MCP discovery responses.

Acceptance criteria

• Core resolves and locks the exact FastMCP 4 beta without compatibility shims.
• All public Core tools register and their schemas remain compatible.
• Modern and legacy client identity paths are covered, including ChatGPT search / fetch.
• No feature depends on Context state surviving an independent request.
• Cloud app callbacks appear in tools/list with _meta.ui.visibility = ["app"] while remaining hidden from model-visible tools at the host boundary.
• The custom Vue app calls stable advertised tool names, not internal hashes.
• Search initiated from the MCP App works in a real ChatGPT host without an approval/unknown-tool block.
• Search preserves authenticated tenant, workspace, and project routing through the ASGI shim.
• Cloud client-info/initialization telemetry works under the modern protocol.
• Core and Cloud targeted tests pass, followed by repository-standard verification gates.

Non-goals for the first Core PR

• Implementing session-backed project selection.
• Adding elicitation/input flows.
• Shipping task-backed bulk reorganization.
• Replacing the semantic/read cache in perf(api): add Redis-cached QUERY and MCP reads #1172.
• Refactoring the Cloud tenant gateway, which is already the correct trust and routing boundary.

References

• FastMCP 4: What's New
• Upgrade from FastMCP 3
• FastMCP sessions
• FastMCP background tasks
• FastMCP response caching
• FastMCP custom HTML apps
• MCP Apps specification
• Cloud MCP Apps PR #1493
• Core caching PR #1172

[claude]

Claude finished @phernandez's task in 1m 14s —— View job

---

Triage complete

• Read issue and gather context
• Classify type/priority/component/complexity
• Check for duplicate issues
• Apply labels
• Post triage summary

Classification

• Type: Feature / enhancement — a phased dependency upgrade (FastMCP 3 → 4 beta) plus a hosted MCP Apps callback-contract fix. Not a bug report against current shipped behavior.
• Component: MCP server core (src/basic_memory/mcp/) and, for Phase 2, the Cloud hosted-app callback wiring (out of this repo's direct scope, tracked via basic-memory-cloud#1493).
• Complexity: Complex. The issue itself proposes a 3-phase breakdown (compatibility port → hosted Apps acceptance → independent feature adoption) specifically to keep regressions isolated — that structure is sound and I'd keep it as the basis for any implementation PRs.
• Priority: Medium. This is a proactive dependency/protocol upgrade with a clear plan rather than an active production outage in this repo. The related live-degradation report (#1175, ChatGPT connector reliability) is a separate, already-filed issue — not a duplicate of this one, but worth linking since Phase 2 here (stable app-only tool names vs. internal hashed addressing) may bear on that failure mode.

Duplicate check: No existing open or closed issue in this repo covers a FastMCP 4 port or the MCP Apps callback-visibility problem. Prior FastMCP-version issues (#468, #463, #164, #944) were all about older breaking changes/pins on FastMCP 2.x/3.x and are resolved/closed — none overlap with this FastMCP 4 beta adoption ask.

Labels applied: enhancement, cloud (repo's actual label set doesn't include mcp-tool/priority/complexity labels, so those aspects are called out in this comment instead).

Note on scope: Per triage instructions, no code changes, branch, or tests were performed — this is analysis and labeling only. The issue's own "Proposed delivery" section is a solid starting point for a follow-up implementation PR whenever the team is ready to scope Phase 1.