diff --git a/docs/docs/reference/telemetry.md b/docs/docs/reference/telemetry.md index b7b33e3176..c3b6a8fda6 100644 --- a/docs/docs/reference/telemetry.md +++ b/docs/docs/reference/telemetry.md @@ -70,8 +70,8 @@ We collect the following categories of events: | `environment_scan_completed` | A `project_scan` finished during onboarding — `has_dbt`, `has_warehouse`, `is_repo`, `connections_found`, and a bounded list of short `degraded` detection keys. No paths, hostnames, or connection details. Emitted only inside an onboarding session, and only once per session, so scans from `/discover` or a model-initiated call are excluded. | | `sample_setup_completed` | The sample dbt project was materialised. `success`, `models`, `tables`, and `reused` — the tool is deliberately re-callable, so this is per invocation. The target path is never sent. | | `activation_menu_shown` | The activation menu was (very likely) rendered. `variant` is `warehouse` or `no_data`. **Derived** — see the note below. | -| `activation_job_selected` / `first_job_completed` | Which activation job the user started and, where observable, finished. Completion is reported only for the job that was actually selected, so the two form a coherent pair. **Derived** — see the note below. | -| `first_prompt_sent` | The user's first typed message in an onboarding session. Slash commands are excluded, so the hidden `/onboard-connect` submission does not count. | +| `activation_job_selected` / `first_job_completed` | Which activation job the user started and, where observable, finished. In onboarding, completion is reported only for the job that was actually selected, so the two form a coherent pair. `first_job_completed` is also sent once per workspace for each machine and Altimate instance, for the first turn in a linked project that had a tool call that didn't fail and finished. **Derived** — see the note below. | +| `first_prompt_sent` | The user's first typed message in an onboarding session, and the first typed message in a linked project, once per workspace for each machine and Altimate instance. Slash commands are excluded, so the hidden `/onboard-connect` submission does not count. | | `onboarding_abandoned` | The CLI exited during a first run without connecting. `last_stage` is the furthest point reached: `started`, `model_picker`, `provider_setup`, `altimate_base_confirm`, or `gateway_auth`. (`connected` is a funnel position but never a `last_stage` — reaching it means the run completed, which is not an abandonment.) Only emitted for a genuine first run — opening `/connect` as an existing user does not enter the funnel, and abandonment after setup completes is out of scope by definition. Emitted on the exit path under a bounded flush, so the measured rate is a lower bound — see [Delivery & Reliability](#delivery--reliability). | | `review_run` | A dbt/SQL review completed or failed — `invocation` (`cli` for `altimate-code review`, `tool` for the `dbt_pr_review` tool), status, duration, and on success the verdict, the pre-gating verdict, mode, risk tier, and finding counts by severity and by category. No file paths, model or column names, finding titles or bodies, SQL, diff content, or repository/branch/PR names. | | `review_post_outcome` | Whether a review was published to GitHub — `not_requested`, `not_attempted`, `target_unresolved`, `full`, `partial`, or `summary_failed`, plus duration. Emitted on the **CLI path only** — the `dbt_pr_review` tool completes reviews but never publishes, so a `review_run` with `invocation: tool` has no post event and that is not a failure. Within the CLI path there is exactly one per **completed** review: a review that failed emits `review_run: failed` and no post event, so absence there means the review failed rather than that an event was lost. `not_attempted` is publication requested but never reached (a bad `--output` path, a stdout write error). No repository, PR, or comment content. | @@ -135,14 +135,16 @@ Other rows gain workspace fields in a linked project: They are therefore inferred from the closest deterministic signals — the menu from the command dispatch or the completed environment scan, the job from the first matching tool or skill invocation that follows. Treat the counts as **lower bounds**, and note two specific gaps: -- The "something else" branch has no tool signature at all and is never counted. -- `first_job_completed` only fires for jobs with a real completion signal. Skill-driven jobs (downstream impact, SQL review, cost) load an instruction bundle and then do their work through other tools, so their completion is not observable and they are absent from this event rather than wrongly counted in it. +- In onboarding, the "something else" branch has no tool signature at all and is never counted. +- In onboarding, `first_job_completed` only fires for jobs with a real completion signal. Skill-driven jobs (downstream impact, SQL review, cost) load an instruction bundle and then do their work through other tools, so their completion is not observable and they are absent from this event rather than wrongly counted in it. + +In a linked project, `first_job_completed` means something broader: a turn with at least one tool call that didn't fail, which finished without an error or a cancel. Nothing says which job that was, so it reports `job` as `something_else`. What has been sent is remembered as one empty file per Altimate instance, workspace and event in the CLI's state directory, outside the project. An onboarding session still sends its own `first_job_completed` for the job it selected, even if the workspace's row was sent before it. ## Delivery & Reliability Telemetry events are buffered in memory and flushed periodically. If a flush fails (e.g., due to a transient network error), events are re-added to the buffer for one retry. On process exit, the CLI performs a final flush to avoid losing events from the current session. -No events are ever written to disk. If the process is killed before the final flush, buffered events are lost. This is by design to minimize on-disk footprint. +No events are ever written to disk. The empty first-event files in the CLI's state directory record only that a workspace's row was sent, and hold no event data. If the process is killed before the final flush, buffered events are lost. This is by design to minimize on-disk footprint. The final flush is **time-bounded** so that quitting never hangs the shell: 2 seconds on the main thread and 5 seconds in the TUI worker. When that budget expires the in-flight request is aborted and its events are dropped rather than retried — a retry would only re-queue them into a buffer that is cleared moments later, to be shipped under the next launch's correlation id. diff --git a/packages/opencode/src/altimate/plugin/onboarding-telemetry.ts b/packages/opencode/src/altimate/plugin/onboarding-telemetry.ts index b249f7fe79..da6043e15b 100644 --- a/packages/opencode/src/altimate/plugin/onboarding-telemetry.ts +++ b/packages/opencode/src/altimate/plugin/onboarding-telemetry.ts @@ -12,6 +12,7 @@ // session loop. import type { Hooks, PluginInput } from "@opencode-ai/plugin" import * as OnboardingTelemetry from "../telemetry/onboarding" +import * as WorkspaceFirstEvents from "../workspace/first-events" // altimate_change start — AI-8398 workspaces trigger. Reaches into the same // EventV2 bridge the server/routes/tui.ts uses to publish TuiEvent.CommandExecute // so the workspace TuiPlugin (packages/opencode/src/plugin/tui/altimate/workspace.tsx) @@ -268,6 +269,10 @@ export async function OnboardingTelemetryPlugin(_input: PluginInput): Promise() +/** Slash commands in sessions with no record. Apart from `sessions`, so an ordinary command still + * can't evict an onboarding session, and capped the same way. */ +const untrackedCommands = new Set() + function record(sessionID: string): SessionRecord { const existing = sessions.get(sessionID) if (existing) return existing @@ -264,12 +268,25 @@ function claim( * the capped map and evict genuine onboarding sessions — after which `isOnboardingSession()` * returns false and the rest of that user's activation events are silently dropped. * - * An untracked session has no onboarding state to protect, so skipping it loses nothing: - * `first_prompt_sent` is gated on `isOnboardingSession` anyway. + * An untracked session has no onboarding state to protect, so it goes in `untrackedCommands`. */ export function noteCommandSubmission(sessionID: string) { const entry = sessions.get(sessionID) - if (entry) entry.commandSubmission = true + if (entry) { + entry.commandSubmission = true + return + } + untrackedCommands.add(sessionID) + if (untrackedCommands.size > MAX_TRACKED_SESSIONS) { + const oldest = untrackedCommands.values().next() + if (!oldest.done) untrackedCommands.delete(oldest.value) + } +} + +/** True (once) if an untracked session's pending user message was command-submitted. A workspace's + * first prompt is counted in any session, so it needs what the record above can't hold. */ +export function consumeUntrackedCommand(sessionID: string): boolean { + return untrackedCommands.delete(sessionID) } /** True (once) if this session's pending user message was command-submitted. */ @@ -334,5 +351,6 @@ export function resetForTest() { abandonedEmitted = false funnelStarted = false sessions.clear() + untrackedCommands.clear() } // altimate_change end diff --git a/packages/opencode/src/altimate/workspace/first-events.ts b/packages/opencode/src/altimate/workspace/first-events.ts new file mode 100644 index 0000000000..a1e79e61f0 --- /dev/null +++ b/packages/opencode/src/altimate/workspace/first-events.ts @@ -0,0 +1,88 @@ +// altimate_change - new file +// +// `first_prompt_sent` and `first_job_completed` for workspaces: once per workspace per machine, from +// any session bound to that workspace. Onboarding sessions keep sending these rows as they always +// did, and mark the workspace as sent, so a machine that onboards in a linked project doesn't send +// them again from a later session. The marks don't hold back an onboarding row: its row for the job +// it selected is sent even when the workspace's row was sent before it. +// +// A job has no finish signal of its own outside `sample_setup`: a skill loads instructions and the +// agent does the work with other tools. So a job here is a turn with at least one tool call that +// didn't fail, ending `completed`, the outcome `task_outcome_signal` reports as `accepted`. It +// reports as `something_else` because nothing says which job it was. +import { createHash } from "node:crypto" +import { mkdir, writeFile } from "node:fs/promises" +import path from "node:path" +import { Global } from "@/global" +import { Log } from "@/altimate/util/log" +import { Telemetry } from "@/altimate/telemetry" +import * as OnboardingTelemetry from "@/altimate/telemetry/onboarding" +import { currentScope } from "./state" + +const log = Log.create({ service: "workspace-first-events" }) + +export type FirstEvent = "first_prompt_sent" | "first_job_completed" + +/** What this process already knows is sent, so a bound turn skips the filesystem after the first. */ +const sent = new Set() + +/** + * True the one time this machine should send `event` for this workspace. + * + * A mark is a file created exclusively, under the app's state directory so nothing lands in the + * user's repository, which makes the claim atomic across the `serve` processes of several windows. + * Workspace ids are per tenant, so the tenant and API are in the key. The credential is not: a second + * person signing in on the machine is still the same machine. + * + * Nothing is claimed while telemetry is off, so a machine that opts back in can still send the row. A + * mark that can't be written claims nothing either, and the next turn tries again. The claim comes + * before the send, so a process killed between the two loses the row rather than duplicating it. + */ +export async function claim(event: FirstEvent, workspaceId: number): Promise { + if (!Telemetry.isEnabled()) return false + const scope = await currentScope().catch(() => null) + if (!scope) return false + const id = createHash("sha256") + .update(JSON.stringify([scope.tenant, scope.apiUrl, workspaceId, event])) + .digest("hex") + .slice(0, 32) + if (sent.has(id)) return false + const dir = path.join(Global.Path.state, "altimate-workspace-first-events") + // Apart from the write: only the exclusive create may answer EEXIST with "already sent". A failure + // here shows up on the write below. + await mkdir(dir, { recursive: true }).catch(() => undefined) + try { + await writeFile(path.join(dir, id), "", { flag: "wx", mode: 0o600 }) + sent.add(id) + return true + } catch (err) { + if ((err as NodeJS.ErrnoException).code === "EEXIST") sent.add(id) + else log.warn("could not record a workspace first event", { event, err: String(err) }) + return false + } +} + +/** A typed prompt: the onboarding row and the workspace row are one row when both apply. */ +export async function promptSent(sessionID: string, workspaceId: number | undefined): Promise { + const onboarding = OnboardingTelemetry.isOnboardingSession(sessionID) && OnboardingTelemetry.claimFirstPrompt(sessionID) + const inWorkspace = workspaceId !== undefined && (await claim("first_prompt_sent", workspaceId)) + if (onboarding || inWorkspace) { + await OnboardingTelemetry.emit({ type: "first_prompt_sent", workspace_id: workspaceId }, sessionID) + } +} + +/** A turn in a workspace that ended `completed` with at least one tool call that didn't fail. */ +export async function jobCompleted(sessionID: string, workspaceId: number | undefined): Promise { + if (workspaceId === undefined) return + if (await claim("first_job_completed", workspaceId)) { + await OnboardingTelemetry.emit( + { type: "first_job_completed", job: "something_else", workspace_id: workspaceId }, + sessionID, + ) + } +} + +/** Test seam: drops this process's memory of the marks, as a new process would start. */ +export function resetForTest() { + sent.clear() +} diff --git a/packages/opencode/src/session/prompt.ts b/packages/opencode/src/session/prompt.ts index 11a455551d..6679346671 100644 --- a/packages/opencode/src/session/prompt.ts +++ b/packages/opencode/src/session/prompt.ts @@ -141,6 +141,7 @@ function workspaceUseFields(sessionID: string, turn: number) { // altimate_change end import { TraceContext } from "@/altimate/observability/trace-context" // altimate_change — client trace per turn import * as OnboardingTelemetry from "@/altimate/telemetry/onboarding" // altimate_change — onboarding funnel +import * as WorkspaceFirstEvents from "@/altimate/workspace/first-events" // altimate_change — a workspace's first prompt and job // altimate_change start — keyless public Zen predicate (flat module) import { isPublicZen } from "@/provider/public-zen" // altimate_change end @@ -1830,6 +1831,12 @@ export namespace SessionPrompt { // per-server timeout. Later catalogs keep the engine tools this request // started with. const boundary = turnBoundary(msgs) + // altimate_change start — a request's slash-command flag is read at its own boundary, so a command + // sent while the loop is busy clears its flag here and doesn't leave it for the next typed message. + const fromCommand = + boundary && + (OnboardingTelemetry.consumeCommandSubmission(sessionID) || OnboardingTelemetry.consumeUntrackedCommand(sessionID)) + // altimate_change end const tools = boundary ? await WorkspaceEngine.atTurnStart(sessionID, catalog) : await catalog() WorkspaceEngine.pinTurnTools(sessionID, boundary, tools) if (boundary) workspaceNotice.settle(sessionID) @@ -1942,25 +1949,20 @@ export namespace SessionPrompt { // - `step === 1` (the enclosing block) is NOT "first message of the session". `step` // is declared inside loop() and loop() runs once per user turn, so this block runs // on every turn. `claimFirstPrompt` is what makes this once-per-session. - // - `isOnboardingSession` scopes it to the funnel. Without it, an onboarding-taxonomy - // event would fire for every session in the product — TUI, `run`, GitHub, API. - // - `consumeCommandSubmission` excludes slash commands, because the scan gate submits + // - `promptSent` scopes it to an onboarding session or a workspace not yet counted on this + // machine. Without that, the row would fire for every session in the product. + // - `fromCommand` excludes slash commands, because the scan gate submits // a hidden `/onboard-connect` as an ordinary user message: it would otherwise be // recorded as the user's first typed prompt in every fresh onboarding. // - a real text part, because a message carrying only attachments (a dragged-in file, // a pasted image) reached this block too and was recorded as the user's first typed // prompt before they had typed anything. Same filter as the intent classifier above. - const fromCommand = OnboardingTelemetry.consumeCommandSubmission(sessionID) const hasUserText = !!userMsg?.parts.some( (p) => p.type === "text" && !p.ignored && !p.synthetic && p.text.trim().length > 0, ) - if ( - !fromCommand && - hasUserText && - OnboardingTelemetry.isOnboardingSession(sessionID) && - OnboardingTelemetry.claimFirstPrompt(sessionID) - ) { - void OnboardingTelemetry.emit({ type: "first_prompt_sent", ...turnWorkspace }, sessionID) + // A subagent's session starts from the command's template or the model's prompt, not a typed one. + if (!fromCommand && hasUserText && !session.parentID) { + await WorkspaceFirstEvents.promptSent(sessionID, boundWorkspace?.datamateId) } // altimate_change end // altimate_change end — session start telemetry @@ -2625,6 +2627,9 @@ export namespace SessionPrompt { ? "abandoned" : "completed" // altimate_change start — emit quality signal, tool chain, and error fingerprint events + if (outcome === "completed" && toolCallCount > toolErrorCount) { + await WorkspaceFirstEvents.jobCompleted(sessionID, boundWorkspace?.datamateId) + } Telemetry.track({ type: "task_outcome_signal", ...turnWorkspace, // altimate_change — this turn's workspace, not the session's current entry diff --git a/packages/opencode/test/altimate/telemetry/onboarding.test.ts b/packages/opencode/test/altimate/telemetry/onboarding.test.ts index 266a394eac..b82d499e6c 100644 --- a/packages/opencode/test/altimate/telemetry/onboarding.test.ts +++ b/packages/opencode/test/altimate/telemetry/onboarding.test.ts @@ -14,6 +14,9 @@ import path from "path" import { Telemetry } from "@/altimate/telemetry" import * as Onboarding from "@/altimate/telemetry/onboarding" import { OnboardingTelemetryPlugin } from "@/altimate/plugin/onboarding-telemetry" +import { AltimateApi } from "@/altimate/api/client" +import * as WorkspaceFirstEvents from "@/altimate/workspace/first-events" +import { withTestStateHome } from "../../fixture/fixture" import type { OnboardingTelemetryEvent } from "@opencode-ai/tui/context/onboarding-telemetry" type Tracked = Telemetry.Event @@ -234,6 +237,31 @@ describe("activation events", () => { expect("workspace_id" in (events.find((e) => e.type === "activation_job_selected") as any)).toBe(false) }) + test("the first job's row marks its workspace, so the workspace's own row isn't sent after it", async () => { + await withTestStateHome(async () => { + spyOn(Telemetry, "isEnabled").mockImplementation(() => true) + spyOn(AltimateApi, "isConfigured").mockImplementation(async () => true) + spyOn(AltimateApi, "getCredentials").mockImplementation( + async () => ({ altimateInstanceName: "acme", altimateUrl: "https://api.test", altimateApiKey: "k" }) as never, + ) + WorkspaceFirstEvents.resetForTest() + captureEvents() + const hooks = await plugin() + await startOnboarding(hooks, "skip") + + await runTool( + hooks, + { tool: "sample_setup", sessionID: SESSION, callID: "c3m", args: {}, workspaceId: 7 }, + { title: "", output: "", metadata: { success: true } }, + ) + await settle() + + expect(await WorkspaceFirstEvents.claim("first_job_completed", 7)).toBe(false) + expect(await WorkspaceFirstEvents.claim("first_job_completed", 8)).toBe(true) + WorkspaceFirstEvents.resetForTest() + }) + }) + test("a failed sample setup counts as selected but not completed", async () => { const events = captureEvents() const hooks = await plugin() diff --git a/packages/opencode/test/altimate/workspace/first-events.test.ts b/packages/opencode/test/altimate/workspace/first-events.test.ts new file mode 100644 index 0000000000..d2272d5cde --- /dev/null +++ b/packages/opencode/test/altimate/workspace/first-events.test.ts @@ -0,0 +1,100 @@ +// altimate_change - new file +// +// The once-per-workspace-per-machine claim behind `first_prompt_sent` and `first_job_completed`. +import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test" +import { mkdtempSync, rmSync, writeFileSync } from "node:fs" +import os from "node:os" +import path from "node:path" +import { Telemetry } from "../../../src/altimate/telemetry" +import { AltimateApi } from "../../../src/altimate/api/client" +import { claim, resetForTest } from "../../../src/altimate/workspace/first-events" + +type Creds = Awaited> + +let stateHome = "" +let restore: (() => void)[] = [] +const originalStateHome = process.env.OPENCODE_TEST_STATE_HOME + +const signInTo = (tenant: string, key = "k") => { + const creds = spyOn(AltimateApi, "getCredentials").mockImplementation( + async () => ({ altimateInstanceName: tenant, altimateUrl: "https://api.test", altimateApiKey: key }) as Creds, + ) + restore.push(() => creds.mockRestore()) +} + +beforeEach(() => { + stateHome = mkdtempSync(path.join(os.tmpdir(), "first-events-state-")) + process.env.OPENCODE_TEST_STATE_HOME = stateHome + resetForTest() + const enabled = spyOn(Telemetry, "isEnabled").mockImplementation(() => true) + const configured = spyOn(AltimateApi, "isConfigured").mockImplementation(async () => true) + restore = [() => enabled.mockRestore(), () => configured.mockRestore()] + signInTo("acme") +}) + +afterEach(() => { + for (const undo of restore) undo() + if (originalStateHome === undefined) delete process.env.OPENCODE_TEST_STATE_HOME + else process.env.OPENCODE_TEST_STATE_HOME = originalStateHome + rmSync(stateHome, { recursive: true, force: true }) +}) + +describe("first-event claim", () => { + test("claims once, in this process and in the next", async () => { + expect(await claim("first_prompt_sent", 7)).toBe(true) + expect(await claim("first_prompt_sent", 7)).toBe(false) + resetForTest() + expect(await claim("first_prompt_sent", 7)).toBe(false) + }) + + test("each event and each workspace has its own claim", async () => { + expect(await claim("first_prompt_sent", 7)).toBe(true) + expect(await claim("first_job_completed", 7)).toBe(true) + expect(await claim("first_prompt_sent", 8)).toBe(true) + }) + + test("two claims racing with no shared memory, as two processes would, get one claim", async () => { + const first = claim("first_prompt_sent", 7) + resetForTest() + const results = await Promise.all([first, claim("first_prompt_sent", 7)]) + expect(results.filter(Boolean)).toHaveLength(1) + }) + + test("the same id in another tenant is another workspace", async () => { + expect(await claim("first_prompt_sent", 7)).toBe(true) + restore.pop()?.() + signInTo("other") + expect(await claim("first_prompt_sent", 7)).toBe(true) + }) + + test("a second person on the machine is the same machine", async () => { + expect(await claim("first_prompt_sent", 7)).toBe(true) + restore.pop()?.() + signInTo("acme", "another-key") + expect(await claim("first_prompt_sent", 7)).toBe(false) + }) + + test("nothing is claimed while telemetry is off, so opting back in can still send", async () => { + const off = spyOn(Telemetry, "isEnabled").mockImplementation(() => false) + expect(await claim("first_prompt_sent", 7)).toBe(false) + off.mockImplementation(() => true) + expect(await claim("first_prompt_sent", 7)).toBe(true) + }) + + test("nothing is claimed with no account", async () => { + restore.pop()?.() + const none = spyOn(AltimateApi, "getCredentials").mockImplementation(async () => { + throw new Error("no credentials") + }) + restore.push(() => none.mockRestore()) + expect(await claim("first_prompt_sent", 7)).toBe(false) + }) + + test("a mark that can't be written claims nothing, and the next turn tries again", async () => { + const blocked = path.join(stateHome, "altimate-workspace-first-events") + writeFileSync(blocked, "not a directory") + expect(await claim("first_prompt_sent", 7)).toBe(false) + rmSync(blocked) + expect(await claim("first_prompt_sent", 7)).toBe(true) + }) +}) diff --git a/packages/opencode/test/session/workspace-first-events-loop.test.ts b/packages/opencode/test/session/workspace-first-events-loop.test.ts new file mode 100644 index 0000000000..c904b12ecd --- /dev/null +++ b/packages/opencode/test/session/workspace-first-events-loop.test.ts @@ -0,0 +1,445 @@ +// altimate_change - new file +/** + * `first_prompt_sent` and `first_job_completed` through the real prompt loop: once per workspace per + * machine, from any session bound to it, whether or not the session is an onboarding one. + */ +import { afterEach, beforeEach, expect, spyOn } from "bun:test" +import { mkdtempSync, rmSync } from "node:fs" +import os from "node:os" +import path from "node:path" +import { Effect, Fiber, Layer } from "effect" +import { LayerNode } from "@opencode-ai/core/effect/layer-node" +import { Session } from "@/session/session" +import { SessionPrompt } from "../../src/session/prompt" +import { SessionSummary } from "../../src/session/summary" +import { Database } from "@opencode-ai/core/database/database" +import { SessionProjector } from "@opencode-ai/core/session/projector" +import { Telemetry } from "../../src/altimate/telemetry" +import * as OnboardingTelemetry from "../../src/altimate/telemetry/onboarding" +import { AltimateApi } from "../../src/altimate/api/client" +import { readLocalBinding, recordApprovedBinding } from "../../src/altimate/workspace/state" +import * as WorkspaceFirstEvents from "../../src/altimate/workspace/first-events" +import { stubEmptySkillList } from "../altimate/workspace/skill-list-fixture" +import { pollWithTimeout, testEffect } from "../lib/effect" +import { TestLLMServer } from "../lib/llm-server" +import { provideTmpdirServerLegacy, withLegacyInstance } from "./legacy-instance" +import { LSP } from "@/lsp/lsp" +import { MCP } from "../../src/mcp" +import { CrossSpawnSpawner } from "@opencode-ai/core/cross-spawn-spawner" +import { RuntimeFlags } from "@/effect/runtime-flags" + +const mcp = Layer.succeed( + MCP.Service, + MCP.Service.of({ + status: () => Effect.succeed({}), + clients: () => Effect.succeed({}), + tools: () => Effect.succeed({}), + prompts: () => Effect.succeed({}), + resources: () => Effect.succeed({}), + add: () => Effect.succeed({ status: { status: "disabled" as const } }), + connect: () => Effect.void, + disconnect: () => Effect.void, + remove: () => Effect.void, + getPrompt: () => Effect.succeed(undefined), + readResource: () => Effect.succeed(undefined), + startAuth: () => Effect.die("unexpected MCP auth"), + authenticate: () => Effect.die("unexpected MCP auth"), + finishAuth: () => Effect.die("unexpected MCP auth"), + removeAuth: () => Effect.void, + supportsOAuth: () => Effect.succeed(false), + hasStoredTokens: () => Effect.succeed(false), + getAuthStatus: () => Effect.succeed("not_authenticated" as const), + entry: () => Effect.succeed(undefined), + listMeta: () => Effect.succeed(undefined), + snapshot: () => Effect.succeed({ tools: {}, meta: undefined }), + }), +) + +const lsp = Layer.succeed( + LSP.Service, + LSP.Service.of({ + init: () => Effect.void, + status: () => Effect.succeed([]), + hasClients: () => Effect.succeed(false), + touchFile: () => Effect.void, + diagnostics: () => Effect.succeed({}), + hover: () => Effect.succeed(undefined), + definition: () => Effect.succeed([]), + references: () => Effect.succeed([]), + implementation: () => Effect.succeed([]), + documentSymbol: () => Effect.succeed([]), + workspaceSymbol: () => Effect.succeed([]), + prepareCallHierarchy: () => Effect.succeed([]), + incomingCalls: () => Effect.succeed([]), + outgoingCalls: () => Effect.succeed([]), + }), +) + +const root = LayerNode.group([ + SessionPrompt.node, + Session.node, + SessionProjector.node, + SessionSummary.node, + Database.node, + CrossSpawnSpawner.node, + LayerNode.make(TestLLMServer.layer, []), +]) +const it = testEffect( + LayerNode.buildLayer(root, { + replacements: [ + LayerNode.replace(MCP.node, mcp), + LayerNode.replace(LSP.node, lsp), + LayerNode.replace(RuntimeFlags.node, RuntimeFlags.layer({ experimentalEventSystem: true })), + ], + }), +) + +const providerCfg = (url: string) => ({ + provider: { + test: { + name: "Test", + id: "test", + env: [], + npm: "@ai-sdk/openai-compatible", + models: { + "test-model": { + id: "test-model", + name: "Test Model", + attachment: false, + reasoning: false, + temperature: false, + tool_call: true, + release_date: "2025-01-01", + limit: { context: 100000, output: 10000 }, + cost: { input: 0, output: 0 }, + options: {}, + }, + }, + options: { apiKey: "test-key", baseURL: url }, + }, + }, + model: "test/test-model", +}) + +const LINKED = 7 + +type Row = { + type: string + workspace_id?: number + job?: string + status?: string + signal?: string + tool_count?: number + had_errors?: boolean +} +type Creds = Awaited> + +let rows: Row[] = [] +let restore: (() => void)[] = [] +let stateHome = "" +const originalStateHome = process.env.OPENCODE_TEST_STATE_HOME + +beforeEach(() => { + rows = [] + stateHome = mkdtempSync(path.join(os.tmpdir(), "first-events-loop-state-")) + process.env.OPENCODE_TEST_STATE_HOME = stateHome + WorkspaceFirstEvents.resetForTest() + OnboardingTelemetry.resetForTest() + const track = spyOn(Telemetry, "track").mockImplementation((event) => void rows.push(event as unknown as Row)) + const enabled = spyOn(Telemetry, "isEnabled").mockImplementation(() => true) + const creds = spyOn(AltimateApi, "getCredentials").mockImplementation( + async () => ({ altimateInstanceName: "acme", altimateUrl: "https://api.test", altimateApiKey: "k" }) as Creds, + ) + const configured = spyOn(AltimateApi, "isConfigured").mockImplementation(async () => true) + restore = [ + () => track.mockRestore(), + () => enabled.mockRestore(), + () => creds.mockRestore(), + () => configured.mockRestore(), + stubEmptySkillList("api.test"), + ] +}) + +afterEach(() => { + for (const undo of restore) undo() + OnboardingTelemetry.resetForTest() + if (originalStateHome === undefined) delete process.env.OPENCODE_TEST_STATE_HOME + else process.env.OPENCODE_TEST_STATE_HOME = originalStateHome + rmSync(stateHome, { recursive: true, force: true }) +}) + +const linkProject = (dir: string) => + Effect.promise(async () => { + await recordApprovedBinding( + dir, + { datamateId: LINKED, datamateName: "project-link", linkedAt: Date.now(), repoRemote: null, projectPath: dir } as never, + { awaitBackfill: true }, + ) + expect((await readLocalBinding(dir))?.datamateId).toBe(LINKED) + }) + +/** + * One typed turn in a new session. `tool` queues a model tool call first, so the turn did some work; + * `failedTool` queues one that fails, a read of a missing file. + */ +const turn = ( + dir: string, + llm: TestLLMServer["Service"], + opts: { tool?: boolean; failedTool?: boolean; before?: (id: string) => void } = {}, +) => + Effect.gen(function* () { + const prompt = yield* SessionPrompt.Service + const sessions = yield* Session.Service + const session = yield* sessions.create({ + title: "a session", + permission: [{ permission: "*", pattern: "*", action: "allow" }], + }) + if (opts.failedTool) yield* llm.tool("read", { filePath: path.join(dir, "missing.txt") }) + if (opts.tool) { + const file = path.join(dir, "note.txt") + yield* Effect.promise(() => Bun.write(file, "note")) + yield* llm.tool("read", { filePath: file }) + } + yield* llm.text("done") + opts.before?.(session.id) + yield* prompt.prompt({ sessionID: session.id, agent: "build", parts: [{ type: "text", text: "do the thing" }] }) + }) + +const named = (type: string) => rows.filter((row) => row.type === type) +const firstPrompts = () => named("first_prompt_sent").map((row) => row.workspace_id) +const firstJobs = () => named("first_job_completed").map((row) => [row.workspace_id, row.job]) + +it.live("a bound session sends both rows, naming the workspace", () => + provideTmpdirServerLegacy( + Effect.fnUntraced(function* ({ dir, llm }) { + yield* linkProject(dir) + yield* turn(dir, llm, { tool: true }) + + expect(firstPrompts()).toEqual([LINKED]) + expect(firstJobs()).toEqual([[LINKED, "something_else"]]) + }), + { git: true, config: providerCfg }, + ), + { timeout: 30_000 }, +) + +it.live("neither row is sent again: not by a later session, nor by a new process", () => + provideTmpdirServerLegacy( + Effect.fnUntraced(function* ({ dir, llm }) { + yield* linkProject(dir) + yield* turn(dir, llm, { tool: true }) + yield* turn(dir, llm, { tool: true }) + WorkspaceFirstEvents.resetForTest() + yield* turn(dir, llm, { tool: true }) + + expect(firstPrompts()).toEqual([LINKED]) + expect(firstJobs()).toEqual([[LINKED, "something_else"]]) + }), + { git: true, config: providerCfg }, + ), + { timeout: 40_000 }, +) + +it.live("a turn with no tool call is a first prompt but not a job, and the job can come later", () => + provideTmpdirServerLegacy( + Effect.fnUntraced(function* ({ dir, llm }) { + yield* linkProject(dir) + yield* turn(dir, llm) + expect(firstPrompts()).toEqual([LINKED]) + expect(firstJobs()).toEqual([]) + + yield* turn(dir, llm, { tool: true }) + expect(firstPrompts()).toEqual([LINKED]) + expect(firstJobs()).toEqual([[LINKED, "something_else"]]) + }), + { git: true, config: providerCfg }, + ), + { timeout: 30_000 }, +) + +it.live("a completed turn whose only tool call failed is not a job, and a later turn with one that works is", () => + provideTmpdirServerLegacy( + Effect.fnUntraced(function* ({ dir, llm }) { + yield* linkProject(dir) + yield* turn(dir, llm, { failedTool: true }) + expect(named("tool_call").map((row) => row.status)).toEqual(["error"]) + expect(named("task_outcome_signal").map((row) => row.signal)).toEqual(["accepted"]) + expect(firstJobs()).toEqual([]) + + yield* turn(dir, llm, { failedTool: true, tool: true }) + expect(firstJobs()).toEqual([[LINKED, "something_else"]]) + }), + { git: true, config: providerCfg }, + ), + { timeout: 30_000 }, +) + +it.live("a turn that ran a tool and then errored is not a job, and leaves the workspace free", () => + provideTmpdirServerLegacy( + Effect.fnUntraced(function* ({ dir, llm }) { + yield* linkProject(dir) + const prompt = yield* SessionPrompt.Service + const sessions = yield* Session.Service + const session = yield* sessions.create({ + title: "errored turn", + permission: [{ permission: "*", pattern: "*", action: "allow" }], + }) + const file = path.join(dir, "note.txt") + yield* Effect.promise(() => Bun.write(file, "note")) + yield* llm.tool("read", { filePath: file }) + yield* llm.error(400, { error: { message: "bad request" } }) + yield* prompt.prompt({ sessionID: session.id, agent: "build", parts: [{ type: "text", text: "do the thing" }] }) + // One tool call that worked, then a provider error ends the turn. + expect(named("task_outcome_signal").map((row) => [row.signal, row.tool_count])).toEqual([["error", 1]]) + expect(named("tool_chain_outcome").map((row) => row.had_errors)).toEqual([false]) + expect(firstJobs()).toEqual([]) + + yield* turn(dir, llm, { tool: true }) + expect(firstJobs()).toEqual([[LINKED, "something_else"]]) + }), + { git: true, config: providerCfg }, + ), + { timeout: 30_000 }, +) + +it.live("a slash command is not a first prompt, and the next typed message is", () => + provideTmpdirServerLegacy( + Effect.fnUntraced(function* ({ dir, llm }) { + yield* linkProject(dir) + yield* turn(dir, llm, { before: (id) => OnboardingTelemetry.noteCommandSubmission(id) }) + expect(firstPrompts()).toEqual([]) + + yield* turn(dir, llm) + expect(firstPrompts()).toEqual([LINKED]) + }), + { git: true, config: providerCfg }, + ), + { timeout: 30_000 }, +) + +it.live("a slash command sent while the session is busy doesn't stop the next typed message counting", () => + provideTmpdirServerLegacy( + Effect.fnUntraced(function* ({ dir, llm }) { + yield* linkProject(dir) + const prompt = yield* SessionPrompt.Service + const sessions = yield* Session.Service + const session = yield* sessions.create({ + title: "queued command", + permission: [{ permission: "*", pattern: "*", action: "allow" }], + }) + const gate = Promise.withResolvers() + yield* llm.hold("first answer", gate.promise) + yield* llm.text("second answer") + + OnboardingTelemetry.noteCommandSubmission(session.id) + const first = yield* prompt + .prompt({ sessionID: session.id, agent: "build", parts: [{ type: "text", text: "first command" }] }) + .pipe(Effect.forkChild) + yield* llm.wait(1) + + // A second command joins the busy loop instead of starting one. + OnboardingTelemetry.noteCommandSubmission(session.id) + const second = yield* withLegacyInstance( + prompt.prompt({ sessionID: session.id, agent: "build", parts: [{ type: "text", text: "queued command" }] }), + ).pipe(Effect.forkChild) + yield* pollWithTimeout( + sessions + .messages({ sessionID: session.id }) + .pipe( + Effect.map((messages) => + messages.some( + (msg) => + msg.info.role === "user" && msg.parts.some((p) => p.type === "text" && p.text === "queued command"), + ) + ? true + : undefined, + ), + ), + "the queued command was never saved", + ) + gate.resolve() + yield* Fiber.join(first) + yield* Fiber.join(second) + const hits = (yield* llm.hits).filter((hit) => !JSON.stringify(hit.body).includes("Generate a title")) + expect(hits).toHaveLength(2) + expect(firstPrompts()).toEqual([]) + + yield* llm.text("third answer") + yield* withLegacyInstance( + prompt.prompt({ sessionID: session.id, agent: "build", parts: [{ type: "text", text: "typed" }] }), + ) + expect(firstPrompts()).toEqual([LINKED]) + }), + { git: true, config: providerCfg }, + ), + { timeout: 30_000 }, +) + +it.live("a subagent's prompt is not a first prompt, and the parent's next typed message is", () => + provideTmpdirServerLegacy( + Effect.fnUntraced(function* ({ dir, llm }) { + yield* linkProject(dir) + const prompt = yield* SessionPrompt.Service + const sessions = yield* Session.Service + const parent = yield* sessions.create({ + title: "parent", + permission: [{ permission: "*", pattern: "*", action: "allow" }], + }) + // What the task tool does for a subtask command or a model's task call: a child session whose + // first message is the command's template or the model's prompt, not something the user typed. + const child = yield* sessions.create({ + parentID: parent.id, + title: "child", + permission: [{ permission: "*", pattern: "*", action: "allow" }], + }) + yield* llm.text("child answer") + yield* prompt.prompt({ sessionID: child.id, agent: "build", parts: [{ type: "text", text: "review the diff" }] }) + expect(firstPrompts()).toEqual([]) + + yield* llm.text("parent answer") + yield* prompt.prompt({ sessionID: parent.id, agent: "build", parts: [{ type: "text", text: "typed" }] }) + expect(firstPrompts()).toEqual([LINKED]) + }), + { git: true, config: providerCfg }, + ), + { timeout: 30_000 }, +) + +it.live("an unbound session sends neither row", () => + provideTmpdirServerLegacy( + Effect.fnUntraced(function* ({ dir, llm }) { + yield* turn(dir, llm, { tool: true }) + + expect(firstPrompts()).toEqual([]) + expect(firstJobs()).toEqual([]) + }), + { git: true, config: providerCfg }, + ), + { timeout: 30_000 }, +) + +it.live("an onboarding session in a bound project sends one row, and a later session none", () => + provideTmpdirServerLegacy( + Effect.fnUntraced(function* ({ dir, llm }) { + yield* linkProject(dir) + yield* turn(dir, llm, { before: (id) => OnboardingTelemetry.markOnboardingSession(id) }) + yield* turn(dir, llm) + + expect(firstPrompts()).toEqual([LINKED]) + }), + { git: true, config: providerCfg }, + ), + { timeout: 30_000 }, +) + +it.live("an onboarding session in an unbound project still sends its row", () => + provideTmpdirServerLegacy( + Effect.fnUntraced(function* ({ dir, llm }) { + yield* turn(dir, llm, { before: (id) => OnboardingTelemetry.markOnboardingSession(id) }) + + expect(firstPrompts()).toEqual([undefined]) + }), + { git: true, config: providerCfg }, + ), + { timeout: 30_000 }, +)