Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -474,6 +474,31 @@ Remove the `mobile-dev` development entry before installing the plugin to avoid
duplicate servers. Public releases use `npm run build:release` and
`npm run package:cursor -- --release`.

### T3 Code local development

T3 Code runs the Claude Code and Codex plugins that its providers have installed.
To test a local build in T3 Code, package both plugins and install them:

```sh
npm run build
npm run package
npm run package:claude
codex plugin marketplace add ./release/marketplace
codex plugin add mobile-dev@mobile-dev-local
claude plugin marketplace add ./release/claude-plugin
claude plugin install mobile-dev@mobile-dev-local
```

Set `MOBILE_DEV_PANEL=browser` on T3 Code's Codex instance, as the
[installation steps](README.md#t3-code) describe. The Codex manifest forwards that
variable to the server, which then serves the panel as a browser page like Claude's
server does: `mobile_open_simulator` has no `ui.resourceUri`, so T3 Code renders no
inline app, and its result carries the `browserPanel` link that the agent opens with
`preview_open`. The shared server in `dist/server.mjs` reads the variable too. An
unknown value logs a warning to the server's stderr and keeps the default placement.
`npm run test:package` checks both placements of the packaged Codex server. After you
reinstall a plugin, start a new thread so that T3 Code starts the new copy.

## Live React UI development

In Codex, run `npm run dev` after installing the plugin. It watches the UI sources and
Expand Down
26 changes: 26 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,32 @@ It sends plugin errors and aggregate measurements to Sentry, as described under
[Privacy and support](#privacy-and-support), and nothing else to Callstack;
profiles go only to the TraceSift service on your Mac.

### T3 Code

[T3 Code](https://github.com/pingdotgg/t3code) runs Claude Code and Codex, so it loads
the Mobile Dev plugin that each of them has installed. On the Mac that runs T3 Code,
install the [Claude Code plugin](#claude-code-and-claude-desktop) for Claude threads
and the [Codex plugin](#codex) for Codex threads. In both, the panel opens in T3 Code's
Browser.

Turn off T3 Code's own device integration while you use Mobile Dev. In
**Settings → Devices**, turn off **Use devices**, which also turns off **Agents can use
devices**. Otherwise T3 Code gives agents its own `device_*` tools, with the same names
as Mobile Dev's, and tells them to prefer those tools.

Codex threads need one setting, because T3 Code renders Codex's panel in the chat
and asks you to confirm each action that the panel takes, such as each tap:

1. Open **Settings → Providers**, then the Codex instance.
2. Under **Variables**, click **Add variable**. Set `MOBILE_DEV_PANEL` to `browser`.
3. Start a new thread.

Ask the agent to open Mobile Dev. It opens the panel with T3 Code's `preview_open`
tool, and T3 Code shows the page as a floating preview. To dock the panel beside the
chat, click the preview's **Open in right panel** button. The agent reads the devices
you select in the panel and the screenshots and screen notes you add from it.
Performance recordings still render as cards in the chat.

### Cursor: manual installation

On an Apple Silicon Mac with Node.js 22.19 or later on `PATH`, obtain the
Expand Down
8 changes: 7 additions & 1 deletion apps/claude/skills/mobile-dev/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,13 @@ result: open it in your in-app browser in the foreground, and reuse a tab that a
shows Mobile Dev. Opening a page can leave the Browser pane hidden behind a card in the
chat, so in the Claude desktop app's Code tab then call `mcp__ccd_view__show_pane` with
`pane: "preview"` (load it with ToolSearch first if it is deferred), so the panel shows
beside the chat without the user clicking Open. Open only that link: the `baseUrl` in device results and other
beside the chat without the user clicking Open. In T3 Code, open it with the
`t3-code` server's `preview_open` (load it with ToolSearch first if it is deferred). T3
shows a page an agent opened as a floating preview that the user can dock beside the
chat, so a result with `visible: false` still counts as shown: do not open it again,
and leave the tab open.
T3's own `mcp__t3-code__device_*` tools drive T3's separate Device panel, so use Mobile
Dev's `device_*` tools for the devices in Mobile Dev's panel. Open only that link: the `baseUrl` in device results and other
`127.0.0.1` addresses belong to Mobile Dev's backends, not to the panel, so never open
or show them. Without a browser, give the user the link. Then read
`mobile_read_panel_context` for the devices the user selected there, and again when
Expand Down
3 changes: 2 additions & 1 deletion apps/codex-plugin/.mcp.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@
"TRACE_SIFT_HOME",
"TRACE_SIFT_COMMAND",
"MOBILE_DEV_TELEMETRY",
"MOBILE_DEV_ENVIRONMENT"
"MOBILE_DEV_ENVIRONMENT",
"MOBILE_DEV_PANEL"
]
}
}
Expand Down
6 changes: 5 additions & 1 deletion apps/codex-plugin/server/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,12 @@ import "@mobile-dev/core/instrument.ts";
import { runStdioServer } from "@mobile-dev/core/stdio-server.ts";
import { createPanel } from "@mobile-dev/core/panel-tools.ts";
import { agentDeviceTools, readAgentDeviceCatalog } from "@mobile-dev/core/agent-device-tools.ts";
import { browserPanel } from "@mobile-dev/core/browser-panel.ts";
import { panelPlacement } from "@mobile-dev/core/panel-placement.ts";
import { codexUpdates } from "./codex-updates.ts";

// Codex's server: the shared server with Mobile Dev's panel and agent-device's commands, plus
// the updater for Codex plugin installs.
await runStdioServer({ panel: createPanel(), features: [agentDeviceTools(readAgentDeviceCatalog()), codexUpdates] });
const inline = panelPlacement("inline") === "inline";
const features = [agentDeviceTools(readAgentDeviceCatalog()), codexUpdates];
await runStdioServer({ panel: createPanel({ inline }), features: inline ? features : [...features, browserPanel({ always: true })] });
2 changes: 2 additions & 0 deletions apps/codex-plugin/test/mcp-launch.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,13 @@ test("the launcher uses Codex's cached Node with spaces, no Node on PATH, and no
const traceSiftHomeForwarded = config.env_vars.includes("TRACE_SIFT_HOME");
const traceSiftCommandForwarded = config.env_vars.includes("TRACE_SIFT_COMMAND");
const profileForwarded = config.env_vars.includes("CODEX_HOME");
const panelForwarded = config.env_vars.includes("MOBILE_DEV_PANEL");
assert.ok(homeForwarded);
assert.ok(telemetryForwarded);
assert.ok(traceSiftHomeForwarded);
assert.ok(traceSiftCommandForwarded);
assert.ok(profileForwarded);
assert.ok(panelForwarded);
const script = 'const args = process.argv.slice(1); const text = JSON.stringify(args); console.log(text);';
const result = launch(fixtureHome, ["--eval", script, "a b", "literal$argument"]);
assert.equal(result.error, undefined);
Expand Down
5 changes: 4 additions & 1 deletion apps/codex-plugin/tooling/mcp-tool-budget.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,15 @@ const { schemaCompactBytes, agentToolBytes, agentTotalBytes } = CODEX_TOOL_LIMIT
const plugin = resolve(process.argv[2] ?? "release/marketplace/plugins/mobile-dev");
const manifest = JSON.parse(await readFile(join(plugin, ".mcp.json"), "utf8"));
const agentPlugin = await access(join(plugin, "plugin.json")).then(() => true, () => false);
const env = { ...process.env, MOBILE_DEV_TELEMETRY: "off" };
// Measure the default inline panel, even from a host set up for the browser panel.
delete env.MOBILE_DEV_PANEL;
let total = 0;
let count = 0;
const oversized = [];
for (const [name, config] of Object.entries(manifest.mcpServers)) {
const client = new Client({ name: "mobile-dev-catalog-budget", version: "1" });
const transport = new StdioClientTransport({ command: process.execPath, args: [config.args[1]], cwd: plugin, stderr: "pipe", env: { ...process.env, MOBILE_DEV_TELEMETRY: "off" } });
const transport = new StdioClientTransport({ command: process.execPath, args: [config.args[1]], cwd: plugin, stderr: "pipe", env });
try {
await client.connect(transport);
const tools = [];
Expand Down
2 changes: 2 additions & 0 deletions apps/codex-plugin/tooling/smoke-codex-plugin.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ try {
await writeFile(catalogPath, catalogText);
const fixtureHome = await createLaunchHome(temporary, process.execPath);
const env = { ...process.env, CODEX_HOME: profile, HOME: fixtureHome, MOBILE_DEV_TELEMETRY: "off" };
// These checks expect the default inline panel, even from a host set up for the browser panel.
delete env.MOBILE_DEV_PANEL;
function codex(args) {
const output = execFileSync(codexCli, args, { env, encoding: "utf8", timeout: 30000 });
return JSON.parse(output);
Expand Down
26 changes: 26 additions & 0 deletions apps/codex-plugin/tooling/smoke-package.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ const temporary = await mkdtemp(join(tmpdir(), "mobile-dev-package-test-"));
const plugin = join(temporary, "mobile-dev");
let transport;
let runtimeTransport;
let browserTransport;

function uiIdentity(text) {
const user = text.match(/name="mobile-dev-user-id" content="(anon_[a-f0-9]{32})"/);
Expand Down Expand Up @@ -43,10 +44,13 @@ try {
assert.ok(serverConfig.env_vars.includes("TRACE_SIFT_HOME"));
assert.ok(serverConfig.env_vars.includes("TRACE_SIFT_COMMAND"));
assert.ok(serverConfig.env_vars.includes("MOBILE_DEV_TELEMETRY"));
assert.ok(serverConfig.env_vars.includes("MOBILE_DEV_PANEL"));
const serverCwd = resolve(plugin, serverConfig.cwd);
// Supply the Codex runtime cache layout with no Node on MCP PATH.
const fixtureHome = await createLaunchHome(temporary, process.execPath);
const serverEnv = { ...process.env, HOME: fixtureHome, PATH: "/usr/bin:/bin:/usr/sbin:/sbin" };
// These checks expect the default inline panel, even from a host set up for the browser panel.
delete serverEnv.MOBILE_DEV_PANEL;
const launcher = serverConfig.args[0];
const nodeVersion = spawnSync(serverConfig.command, [launcher, "--version"], { cwd: serverCwd, env: serverEnv, encoding: "utf8" });
assert.equal(nodeVersion.error, undefined);
Expand Down Expand Up @@ -241,7 +245,29 @@ try {
}
assert.equal(stopped, true, "The bundled backend should stop when MCP closes.");
console.log("MCP shutdown stopped the bundled backend. No simulator was booted or changed.");
// T3 Code users set MOBILE_DEV_PANEL=browser, so the panel opens on a loopback page beside the chat.
browserTransport = new StdioClientTransport({
command: serverConfig.command, args: serverConfig.args, cwd: serverCwd, stderr: "pipe", env: { ...serverEnv, MOBILE_DEV_PANEL: "browser" },
});
const browser = new Client({ name: "mobile-dev-package-browser", version: "1" });
await browser.connect(browserTransport);
const browserTools = await browser.listTools();
const browserEntrypoint = browserTools.tools.find(tool => tool.name === "mobile_open_simulator");
assert.equal(browserEntrypoint._meta.ui?.resourceUri, undefined, "A browser panel must not render in the chat.");
assert.equal(browserEntrypoint._meta["ui/resourceUri"], undefined, "A browser panel must not render in the chat.");
assert.equal(browserEntrypoint._meta["mobile-dev/panel"].resourceUri, `ui://mobile-dev/${manifest.version}/simulator.html`);
assert.ok(browserTools.tools.some(tool => tool.name === "mobile_read_panel_context"));
await access(join(plugin, "dist/browser-panel.html"));
const browserOpening = await browser.callTool({ name: "mobile_open_simulator", arguments: {} }, undefined, { timeout: 30000 });
assert.ok(browserOpening.structuredContent?.browserPanel, JSON.stringify(browserOpening.content));
const browserPanelUrl = new URL(browserOpening.structuredContent.browserPanel.url);
assert.equal(browserPanelUrl.hostname, "127.0.0.1");
const browserPage = await fetch(browserPanelUrl);
assert.equal(browserPage.status, 200);
await browser.close();
console.log("With MOBILE_DEV_PANEL=browser, the packaged server names its panel for the browser and serves the page on loopback.");
} finally {
await browserTransport?.close();
await runtimeTransport?.close();
await transport?.close();
await rm(temporary, { recursive: true, force: true });
Expand Down
6 changes: 5 additions & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,11 @@ A host adds to the shared pieces without changing them:
resource. Claude's entry (`apps/claude/server`) passes the panel with `inline: false`,
so even MCP Apps hosts such as the desktop app's Code tab render none in the chat, and
the browser panel feature with `always: true`, which shows it beside the chat in a
browser; it adds agent-device's commands. Every host lists simulators for logs, CPU and device choices with
browser; it adds agent-device's commands. `MOBILE_DEV_PANEL=browser`
(`packages/core/src/panel-placement.ts`) gives the Codex and shared entries Claude's setup,
and recording cards then offer no workspace. MCP Apps hosts advertise the same capabilities
whether or not an inline panel suits them. T3 Code, for example, asks the user to confirm every
panel tool call that is not read-only, so its Codex threads need `browser`. Every host lists simulators for logs, CPU and device choices with
simctl, without starting Baguette.
- **Runtime:** a host's `plugin-host.json` can leave out the panel's parts of `dist/`
(`"panel": false`, see `PANEL_PARTS` in `tooling/host-runtime.mjs`), download them
Expand Down
3 changes: 2 additions & 1 deletion docs/telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@ Behavior never depends on this value; it only attributes measurements and errors
Cursor sessions were reported as `other`. Calls before connecting carry no host.
- The server labels itself from the connecting MCP client's name on initialize and
reports `other` until then.
- In hosts that render no MCP Apps panel, such as Claude Code, the panel can open in
- In hosts that render no MCP Apps panel, such as Claude Code, and with
`MOBILE_DEV_PANEL=browser`, as in T3 Code's Codex threads, the panel can open in
a browser beside the chat. Its page names itself with the server's label for the
chat host, so the panel is attributed to that host. Its requests join the host's
server session with the same trace propagation; the browser page adds no
Expand Down
2 changes: 1 addition & 1 deletion packages/core/src/browser-panel.ts
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ export function browserPanel({ template = PAGE, always = false }: BrowserPanelOp
page ??= startPanelPage(connections, host, template);
const opened = await page;
const url = opened.url(tool, target);
const instructions = `Mobile Dev's panel is a page to show beside the chat, not in it, and it is not open yet. Show it to the user now: open ${url} in your in-app browser or side panel, in the foreground, so the page is visible without another click. Opening a page can leave the browser pane hidden: when your host has a tool that brings its browser or preview pane into view, such as a show_pane tool for its preview pane, load it if needed and call it right after. Reuse a tab that already shows Mobile Dev. Open no other address for it, such as the backend's baseUrl. If you cannot open pages, give the user the link. The page shares the user's devices and what they add to chat through mobile_read_panel_context; read it before device work.`;
const instructions = `Mobile Dev's panel is a page to show beside the chat, not in it, and it is not open yet. Show it to the user now: open ${url} in your in-app browser or side panel, in the foreground, so the page is visible without another click. This page is how Mobile Dev shows its panel, so opening it is part of opening the panel. Use your host's own browser or preview tool, and load it first if it is deferred. Never open the page with a computer-use tool that drives the screen. Opening a page can leave the browser pane hidden: when your host has a tool that brings its browser or preview pane into view, such as a show_pane tool for its preview pane, load it if needed and call it right after. A host can also show the page as a floating preview and report it as not visible. That still counts as shown, so do not open it again. Leave the tab open after you reply. Reuse a tab that already shows Mobile Dev. Open no other address for it, such as the backend's baseUrl. If you cannot open pages, give the user the link. The page shares the user's devices and what they add to chat through mobile_read_panel_context; read it before device work.`;
return { url, instructions };
});
// A browser has no model context of its own, so the agent asks the server for it.
Expand Down
6 changes: 3 additions & 3 deletions packages/core/src/device-apps/foreground.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import { fileURLToPath } from "node:url";
import { z } from "zod";
import { adbPath } from "../native-logs.ts";
import { baguetteEnvironment, BAGUETTE_RUNTIME_TIMEOUT_MS } from "../baguette-runtime.ts";
import { panelRuntimePath } from "../panel-runtime.ts";
import type { ForegroundApp } from "@mobile-dev/shared/device-apps.ts";
import { ExpectedOperationError } from "@mobile-dev/shared/error-reporting.ts";

Expand All @@ -29,9 +30,8 @@ export type SimulatorForeground = (deviceId: string, signal?: AbortSignal) => Pr
/** Servers without the panel ship no Baguette, so a simulator's foreground app is unknown. */
export const unknownSimulatorForeground: SimulatorForeground = async () => null;

export async function foregroundSimulatorPid(deviceId: string, signal?: AbortSignal,
helper = new URL("./baguette/Baguette", import.meta.url), runtime = baguetteEnvironment): Promise<number | null> {
const path = fileURLToPath(helper);
export async function foregroundSimulatorPid(deviceId: string, signal?: AbortSignal, runtime = baguetteEnvironment): Promise<number | null> {
const path = await panelRuntimePath("baguette/Baguette");
const cancellation = signal ?? AbortSignal.timeout(10000);
const startedAt = performance.now();
let environment: NodeJS.ProcessEnv;
Expand Down
4 changes: 3 additions & 1 deletion packages/core/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@ import { runStdioServer } from "./stdio-server.ts";
import { createPanel } from "./panel-tools.ts";
import { agentDeviceTools, readAgentDeviceCatalog } from "./agent-device-tools.ts";
import { browserPanel } from "./browser-panel.ts";
import { panelPlacement } from "./panel-placement.ts";

// The shared server with Mobile Dev's panel and agent-device's commands, as Cursor and other
// MCP hosts run it. Hosts that render no MCP Apps panel can show it in a browser beside the chat.
await runStdioServer({ panel: createPanel(), features: [agentDeviceTools(readAgentDeviceCatalog()), browserPanel()] });
const inline = panelPlacement("inline") === "inline";
await runStdioServer({ panel: createPanel({ inline }), features: [agentDeviceTools(readAgentDeviceCatalog()), browserPanel({ always: !inline })] });
13 changes: 13 additions & 0 deletions packages/core/src/panel-placement.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
export type PanelPlacement = "inline" | "browser";

/**
* MCP Apps hosts advertise the same capabilities whether or not an inline panel suits them,
* such as T3 Code, which asks the user to confirm each app tool call that is not read-only,
* so only the user can choose the placement.
*/
export function panelPlacement(fallback: PanelPlacement, env: NodeJS.ProcessEnv = process.env): PanelPlacement {
const requested = env.MOBILE_DEV_PANEL?.trim().toLowerCase();
if (requested === "inline" || requested === "browser") return requested;
if (requested) console.error(`[mobile-dev] Ignoring MOBILE_DEV_PANEL=${JSON.stringify(env.MOBILE_DEV_PANEL)}; it accepts "inline" or "browser". Using "${fallback}".`);
return fallback;
}
3 changes: 3 additions & 0 deletions packages/core/src/panel-tools.ts
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,8 @@ const INSTRUCTIONS = "For mobile app development, open mobile_open_simulator bef

/** Mobile Dev's own device views, which a host adds to the server it runs (see createPanel). */
export type Panel = {
/** Whether the panel renders in the chat, where it can also open recordings in the workspace. */
readonly inline: boolean;
instructions: string;
simulatorForeground: SimulatorForeground;
register(server: McpServer, shared: PanelShared): () => Promise<void>;
Expand Down Expand Up @@ -112,6 +114,7 @@ export type PanelOptions = {
*/
export function createPanel({ baguette = new Baguette(), simulatorInput, copyScreenshot = copyPNGToClipboard, inline = true }: PanelOptions = {}): Panel {
return {
inline,
instructions: INSTRUCTIONS,
simulatorForeground: foregroundSimulatorPid,
register: (server, shared) => registerPanel(server, shared, {
Expand Down
Loading
Loading