Skip to content

fix(telemetry): opt-in default + picker funnel events#3308

Merged
louisgv merged 1 commit intoOpenRouterTeam:mainfrom
AhmedTMM:fix/funnel-earlier
Apr 15, 2026
Merged

fix(telemetry): opt-in default + picker funnel events#3308
louisgv merged 1 commit intoOpenRouterTeam:mainfrom
AhmedTMM:fix/funnel-earlier

Conversation

@AhmedTMM
Copy link
Copy Markdown
Collaborator

Summary

Fixes two bugs discovered after #3305 merged and data started flowing:

  1. Test pollution — the onboarding funnel filled up with CI runs tagged `agent=testagent` because `orchestrate.test.ts` imports `runOrchestration` directly (never calls `initTelemetry`), and `_enabled` defaulted to `true` in the module. So `captureEvent` happily fired real events at PostHog on every test run.
  2. Funnel started too late — the `funnel_*` events were instrumented in `runOrchestration`, which only runs after the interactive picker completes. Users who bail at agent/cloud/setup-options/name prompts were invisible — yet that's exactly where real drop-off happens, and exactly what Alex's DigitalOcean-trial user bailed at.

Fix 1 — opt-in default

`telemetry.ts`:

  • Default `_enabled = false`. Nothing fires until `initTelemetry` is explicitly called. Production (`index.ts`) calls it; tests that need telemetry (`telemetry.test.ts`) call it with `BUN_ENV`/`NODE_ENV` cleared.
  • Belt-and-suspenders: `initTelemetry` now short-circuits when `BUN_ENV === "test"` or `NODE_ENV === "test"`, so even if future code calls it from a test context, events stay local.

Fix 2 — picker funnel events

New events fired in every entry path before `runOrchestration` is called:

Event Properties
`spawn_launched` `mode`: `interactive` / `agent_interactive` / `direct` / `headless`
`menu_shown` / `menu_selected` / `menu_cancelled` only when user has prior spawns
`agent_picker_shown`
`agent_selected` `agent` — also sets telemetry context
`cloud_picker_shown`
`cloud_selected` `cloud` — also sets telemetry context
`preflight_passed`
`setup_options_shown`
`setup_options_selected` `step_count`
`name_prompt_shown`
`name_entered`
`picker_completed` fires right before `execScript` / `runOrchestration` hand-off

Wired into:

  • `commands/interactive.ts` — `cmdInteractive` (full picker) + `cmdAgentInteractive` (short-form `spawn claude`)
  • `commands/run.ts` — `cmdRun` (direct `spawn `) and `cmdRunHeadless` (only `spawn_launched` since there's no picker)

`runOrchestration`'s existing `funnel_*` events continue to fire unchanged, appended to the end of the funnel. The full sequence in PostHog is now:

```
spawn_launched
→ agent_selected
→ cloud_selected
→ preflight_passed
→ setup_options_selected
→ name_entered
→ picker_completed
→ funnel_started (existing)
→ funnel_cloud_authed
→ funnel_credentials_ready
→ funnel_vm_ready
→ funnel_install_completed
→ funnel_configure_completed
→ funnel_prelaunch_completed
→ funnel_handoff
```

Dashboard follow-up (manual, not in this PR)

After merge, the PostHog funnel insight should be rebuilt to start from `spawn_launched` (not `funnel_started`). The existing `testagent` events from before this fix can't be deleted, but they're trivially filterable — add `agent != testagent` as a property filter on the insight.

Tests

  • `telemetry.test.ts` — 2 new env-guard tests (`BUN_ENV=test`, `NODE_ENV=test`) that lock in the CI-safety regression. `beforeEach` updated to save/restore `BUN_ENV` and `NODE_ENV` so existing tests can still exercise `initTelemetry`.
  • Full suite: 2131/2131 pass (was 2129, +2 new), biome 0 errors on 187 files.

Version

Bumps 1.0.12 → 1.0.13. Patch bump — auto-propagates under the #3296 policy so the fix reaches users on their next spawn run.

Closes the regression introduced by #3305.

Two bugs from the OpenRouterTeam#3305 rollout:

1. Test pollution: orchestrate.test.ts imports runOrchestration directly
   and never calls initTelemetry, but _enabled defaulted to true in the
   module so captureEvent happily fired real events at PostHog tagged
   agent=testagent. The onboarding funnel filled up with CI fixture data.

2. Funnel started too late: funnel_* events fired inside runOrchestration,
   which is only called AFTER the interactive picker completes. Users who
   bail at the agent/cloud/setup-options/name prompts were invisible —
   yet that's exactly where real drop-off happens.

Fix 1 — telemetry.ts:
  - Default _enabled = false. Nothing fires until initTelemetry is
    explicitly called. Production (index.ts) calls it; tests that need
    telemetry (telemetry.test.ts) call it with BUN_ENV/NODE_ENV cleared.
  - Belt-and-suspenders: initTelemetry now short-circuits when
    BUN_ENV === "test" || NODE_ENV === "test", so even if future code
    calls it from a test context, events stay local.

Fix 2 — picker instrumentation:
  New events fired before runOrchestration in every entry path:

    spawn_launched         { mode: interactive | agent_interactive | direct | headless }
    menu_shown / menu_selected / menu_cancelled   (only when user has prior spawns)
    agent_picker_shown
    agent_selected         { agent }     — also sets telemetry context
    cloud_picker_shown
    cloud_selected         { cloud }     — also sets telemetry context
    preflight_passed
    setup_options_shown
    setup_options_selected { step_count }
    name_prompt_shown
    name_entered
    picker_completed

  Wired into:
    commands/interactive.ts  cmdInteractive + cmdAgentInteractive
    commands/run.ts          cmdRun (direct `spawn <agent> <cloud>`)
                             cmdRunHeadless (only spawn_launched)

  runOrchestration's existing funnel_* events continue to fire unchanged.
  The final funnel in PostHog:
    spawn_launched → agent_selected → cloud_selected → preflight_passed
    → setup_options_selected → name_entered → picker_completed
    → funnel_started → funnel_cloud_authed → funnel_credentials_ready
    → funnel_vm_ready → funnel_install_completed → funnel_configure_completed
    → funnel_prelaunch_completed → funnel_handoff

Tests:
- telemetry.test.ts: 2 new env-guard tests (BUN_ENV, NODE_ENV), plus
  updated beforeEach to clear both env vars so existing tests still
  exercise initTelemetry.
- Full suite: 2131/2131 pass, biome 0 errors.

Bumps 1.0.12 -> 1.0.13 (patch — auto-propagates under OpenRouterTeam#3296 policy).
Copy link
Copy Markdown
Member

@louisgv louisgv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security Review

Verdict: APPROVED
Commit: 72c0cf0

Findings

No security issues found. This PR improves telemetry safety:

  • Opt-in default: Telemetry starts disabled (_enabled = false) until initTelemetry() is explicitly called
  • Test environment guard: Blocks telemetry in CI (BUN_ENV=test, NODE_ENV=test) to prevent test fixtures polluting production analytics
  • PII protection maintained: All new captureEvent() calls only send non-sensitive metadata (agent/cloud names, mode strings, counts)
  • Existing scrubbing active: String values still pass through the scrubber that redacts API keys, tokens, emails, IPs, paths

Tests

  • bun test: PASS (2131 tests, all green)
  • bash -n: N/A (no shell scripts changed)
  • Test coverage: New tests verify the BUN_ENV=test and NODE_ENV=test guards work correctly

-- security/pr-reviewer

@louisgv louisgv added the security-approved Security review approved label Apr 15, 2026
@louisgv louisgv merged commit a179fdb into OpenRouterTeam:main Apr 15, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

security-approved Security review approved

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants