Skip to content

feat(llm): attribute and tag OpenRouter requests - #341

Merged
Makisuo merged 1 commit into
mainfrom
feat/openrouter-app-attribution
Aug 4, 2026
Merged

feat(llm): attribute and tag OpenRouter requests#341
Makisuo merged 1 commit into
mainfrom
feat/openrouter-app-attribution

Conversation

@Makisuo

@Makisuo Makisuo commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Why

Every OpenRouter request Maple makes is anonymous today. platform/Llm.ts calls OpenRouter.configure({ apiKey }) with nothing else, and slack-agent calls createOpenRouter({ apiKey }) the same way. So:

  • The OpenRouter dashboard is one undifferentiated pile of spend — no way to tell interactive chat from AI triage from the Slack agent, or to attribute cost to an org.
  • Maple has no app page on openrouter.ai. OpenRouter only creates one for traffic carrying an HTTP-Referer header, so Maple is invisible on /rankings and gets none of the per-app analytics at /apps.
  • docs/openrouter-tracing.md claims both already work and points at apps/chat-agent/src/lib/openrouter.ts, deleted in c3c7cc1a8.

What changed

apps/api/src/platform/Llm.ts — the single seam, covers chat and triage in one edit:

  • HTTP-Referer: https://maple.dev + X-Title: Maple on every OpenRouter call.
  • New optional LlmCallTags on resolveTriageModel(env, tags), folded into the request body as route defaults: user: <orgId>, session_id, trace: { trace_name }. Because they are route defaults, every generate/stream/generateObject made with that model inherits them — including triage's final structured pass in triage-agent.ts — without each call site threading anything through.
  • The Workers AI branch ignores tags entirely. These are OpenRouter's body fields and must never be posted to Cloudflare.

Call sitesturn-runner.ts tags chat + orgId + chat session id; AiTriageWorkflow.run.ts tags ai-triage + orgId + triage_<kind>_<incidentId> (incidentId was already on the input, just unused).

apps/slack-agent/agent/agent.ts — the provider's first-class appUrl/appName (present in the pinned 3.0.0) plus a static extraBody: { trace: { trace_name: "slack" } }. Same referer as apps/api on purpose: the referer is the app's identity, so a second value would mint a second app entry and split the rankings.

docs/openrouter-tracing.md — rewritten against the live docs (verified 2026-08-04).

Reviewer notes

  • No lib/llm changes. ModelOptions already extends RouteDefaultsInput, which carries both headers and http; mergeRouteDefaults folds them onto LLMRequest.http; and user/session_id/trace are not in PROTOCOL_BODY_OVERLAY_DENYLIST (that only guards protocol-owned generation fields). So nothing here is at risk from bun run llm:sync, and no cassettes move.
  • Where each signal lands is not uniform, which is why all three are sent: user appears on /activity and in exports (OpenRouter folds it into a hashed identity and never forwards it raw upstream); session_id groups a conversation and makes OpenRouter route it stickily, so prompt caches hit better; trace is Broadcast-only and will not show in the OpenRouter dashboard until a Broadcast destination is configured.
  • MCP evals stay unattributed (__evals__/model.ts) so CI spend stays out of the product's numbers — only its stale comment was fixed.
  • Not fixed here: canReachModel() in AiTriageWorkflow.run.ts gates the OpenRouter path on Cloudflare credentials and never checks OPENROUTER_API_KEY. Real pre-existing bug, unrelated to this change.

Testing

New apps/api/src/platform/Llm.test.ts swaps FetchHttpClient.Fetch for a capture and asserts the actual outgoing request. LLMClient.prepare is no good here — it returns the protocol body, built before the http.body overlay, so it cannot see the tags at all. The fake responds 400 (non-retryable) to keep each case to one request with no backoff.

Covers: attribution headers present; user/session_id/trace present; session_id omitted when there is no session; over-long session id truncated to OpenRouter's 256-char limit; and none of it on the Workers AI path.

  • bun run --cwd apps/api vitest run src/platform/Llm.test.ts src/chat → 27 passed
  • bun typecheck → clean
  • bunx tsc --noEmit in apps/slack-agent → exit 0

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Every OpenRouter request Maple made was anonymous: the dashboard showed one
undifferentiated pile of spend with no way to tell chat from AI triage from the
Slack agent, or to attribute cost to an org. Maple also had no app page on
openrouter.ai at all, since that is only created for traffic carrying an
HTTP-Referer header.

Attribution rides on every OpenRouter call now (HTTP-Referer: https://maple.dev,
X-Title: Maple), and resolveTriageModel takes optional LlmCallTags that it folds
into the request body as route defaults - user (org id), session_id, and
trace.trace_name. Being route defaults, every generate/stream/generateObject made
with that model inherits them, including triage's final structured pass, without
each call site threading anything through. The Workers AI branch ignores the tags
entirely: those are OpenRouter's body fields.

apps/slack-agent uses the provider's first-class appUrl/appName plus a static
trace_name, deliberately with the same referer as apps/api - the referer is the
app's identity, so a second value would mint a second app entry and split the
rankings. MCP evals stay unattributed so CI spend stays out of the product's
numbers.

Nothing in lib/llm changed: ModelOptions already carries headers and http, and
user/session_id/trace are not in the transport's body-overlay denylist.

docs/openrouter-tracing.md described headers and a BYOK table that no longer
exist and pointed at files deleted in c3c7cc1; rewritten to match reality.
@pullfrog

pullfrog Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Your Pullfrog Router balance is exhausted.

You have a payment method on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer.

Top up balance → · Enable auto-reload →

Pullfrog  | Rerun failed job ➔View workflow run | via Pullfrog | Using Claude Opus𝕏

@Makisuo
Makisuo merged commit ccbefc9 into main Aug 4, 2026
11 of 12 checks passed
@Makisuo
Makisuo deleted the feat/openrouter-app-attribution branch August 4, 2026 17:52
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown

🍁 Maple PR preview

Note

Preview resources were removed when this pull request closed.

Final commit 2161141 · View workflow run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant