feat(llm): attribute and tag OpenRouter requests - #341
Merged
Conversation
Every OpenRouter request Maple made was anonymous: the dashboard showed one undifferentiated pile of spend with no way to tell chat from AI triage from the Slack agent, or to attribute cost to an org. Maple also had no app page on openrouter.ai at all, since that is only created for traffic carrying an HTTP-Referer header. Attribution rides on every OpenRouter call now (HTTP-Referer: https://maple.dev, X-Title: Maple), and resolveTriageModel takes optional LlmCallTags that it folds into the request body as route defaults - user (org id), session_id, and trace.trace_name. Being route defaults, every generate/stream/generateObject made with that model inherits them, including triage's final structured pass, without each call site threading anything through. The Workers AI branch ignores the tags entirely: those are OpenRouter's body fields. apps/slack-agent uses the provider's first-class appUrl/appName plus a static trace_name, deliberately with the same referer as apps/api - the referer is the app's identity, so a second value would mint a second app entry and split the rankings. MCP evals stay unattributed so CI spend stays out of the product's numbers. Nothing in lib/llm changed: ModelOptions already carries headers and http, and user/session_id/trace are not in the transport's body-overlay denylist. docs/openrouter-tracing.md described headers and a BYOK table that no longer exist and pointed at files deleted in c3c7cc1; rewritten to match reality.
Contributor
|
Your Pullfrog Router balance is exhausted. You have a payment method on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer. Top up balance → · Enable auto-reload →
|
🍁 Maple PR previewNote Preview resources were removed when this pull request closed. Final commit |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Why
Every OpenRouter request Maple makes is anonymous today.
platform/Llm.tscallsOpenRouter.configure({ apiKey })with nothing else, andslack-agentcallscreateOpenRouter({ apiKey })the same way. So:HTTP-Refererheader, so Maple is invisible on/rankingsand gets none of the per-app analytics at/apps.docs/openrouter-tracing.mdclaims both already work and points atapps/chat-agent/src/lib/openrouter.ts, deleted inc3c7cc1a8.What changed
apps/api/src/platform/Llm.ts— the single seam, covers chat and triage in one edit:HTTP-Referer: https://maple.dev+X-Title: Mapleon every OpenRouter call.LlmCallTagsonresolveTriageModel(env, tags), folded into the request body as route defaults:user: <orgId>,session_id,trace: { trace_name }. Because they are route defaults, everygenerate/stream/generateObjectmade with that model inherits them — including triage's final structured pass intriage-agent.ts— without each call site threading anything through.tagsentirely. These are OpenRouter's body fields and must never be posted to Cloudflare.Call sites —
turn-runner.tstagschat+ orgId + chat session id;AiTriageWorkflow.run.tstagsai-triage+ orgId +triage_<kind>_<incidentId>(incidentIdwas already on the input, just unused).apps/slack-agent/agent/agent.ts— the provider's first-classappUrl/appName(present in the pinned 3.0.0) plus a staticextraBody: { trace: { trace_name: "slack" } }. Same referer asapps/apion purpose: the referer is the app's identity, so a second value would mint a second app entry and split the rankings.docs/openrouter-tracing.md— rewritten against the live docs (verified 2026-08-04).Reviewer notes
lib/llmchanges.ModelOptionsalready extendsRouteDefaultsInput, which carries bothheadersandhttp;mergeRouteDefaultsfolds them ontoLLMRequest.http; anduser/session_id/traceare not inPROTOCOL_BODY_OVERLAY_DENYLIST(that only guards protocol-owned generation fields). So nothing here is at risk frombun run llm:sync, and no cassettes move.userappears on/activityand in exports (OpenRouter folds it into a hashed identity and never forwards it raw upstream);session_idgroups a conversation and makes OpenRouter route it stickily, so prompt caches hit better;traceis Broadcast-only and will not show in the OpenRouter dashboard until a Broadcast destination is configured.__evals__/model.ts) so CI spend stays out of the product's numbers — only its stale comment was fixed.canReachModel()inAiTriageWorkflow.run.tsgates the OpenRouter path on Cloudflare credentials and never checksOPENROUTER_API_KEY. Real pre-existing bug, unrelated to this change.Testing
New
apps/api/src/platform/Llm.test.tsswapsFetchHttpClient.Fetchfor a capture and asserts the actual outgoing request.LLMClient.prepareis no good here — it returns the protocol body, built before thehttp.bodyoverlay, so it cannot see the tags at all. The fake responds 400 (non-retryable) to keep each case to one request with no backoff.Covers: attribution headers present;
user/session_id/tracepresent;session_idomitted when there is no session; over-long session id truncated to OpenRouter's 256-char limit; and none of it on the Workers AI path.bun run --cwd apps/api vitest run src/platform/Llm.test.ts src/chat→ 27 passedbun typecheck→ cleanbunx tsc --noEmitinapps/slack-agent→ exit 0Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.