[HDX-5162] LLM observability dashboard, span chat view, and sessions - #2990
Open
wrn14897 wants to merge 4 commits into
Open
[HDX-5162] LLM observability dashboard, span chat view, and sessions#2990wrn14897 wants to merge 4 commits into
wrn14897 wants to merge 4 commits into
Conversation
…s, and /llm dashboard
Adds read-time, schema-agnostic LLM observability on top of existing trace
and log data. No ingestion changes: everything derives from span/log
attribute maps at query time, so it works retroactively on already-ingested
telemetry.
- Normalization lib (packages/app/src/llm/lib): detects LLM spans and
normalizes model, provider, token usage (incl. cached/reasoning), cost,
session ids, TTFT, and chat messages across four instrumentation dialects:
OTel GenAI semconv (attribute- and event-based), OpenLLMetry,
OpenInference, and the Vercel AI SDK. Includes real-world variants
observed from opencode and Claude Code telemetry (whole-string
llm.input_messages, camelCase ai.usage.*, flat token/cost keys,
bracketed model ids like claude-opus-5[1m]).
- Span side panel: an LLM tab renders the normalized conversation (roles,
markdown, tool calls) with a usage/cost summary; the Overview tab gains an
LLM section; trace waterfall labels LLM spans with model + token count.
- Cost estimation: bundled model price catalog (adapted from Langfuse's
MIT-licensed price list) with regex matching for provider/Bedrock/Vertex
id flavors; an instrumentation-provided cost attribute always wins.
- /llm preset dashboard (Overview | Sessions | Search):
- Overview: KPI tiles (calls, tokens, est. cost, avg cost/call, cache hit
rate, error rate), calls/error trends, token split
(uncached/cached/output/reasoning), cost by model, finish reasons,
models/services/users/error tables, latency heatmap, p95 by model,
TTFT, and tool analytics.
- Sessions: activity grouped by the cross-dialect session id
(gen_ai.conversation.id, session.id, ai.telemetry.metadata.sessionId)
with a drawer timeline; span attributes load lazily per expanded call
since agent SDKs stamp full conversation history on every span
(~48 MiB -> ~20 KiB list payload).
- Search: side-by-side LLM trace span and log event tables, correlating
signals for instrumentations that emit session ids without trace
context.
- Aggregations gate token/cost sums on authoritative usage reporters so SDK
wrapper spans don't double count.
Known limitations: bundled prices go stale between releases, and apps that
double-instrument (e.g. opencode emitting both OpenInference and Vercel AI
spans per call) still double count in sums.
🦋 Changeset detectedLatest commit: c37f5fc The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
github-actions
Bot
added
the
review/tier-4
Contributor
🔴 Tier 4 — CriticalTouches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI. Why this tier:
Review process: Deep review from a domain expert. Synchronous walkthrough may be required. Stats
|
Contributor
Greptile SummaryThis PR adds schema-agnostic LLM telemetry normalization, conversation rendering, cost estimation, session exploration, and a preset LLM dashboard.
Confidence Score: 5/5The PR appears safe to merge because no blocking failure remains. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| packages/app/src/llm/lib/extract.ts | Coordinates dialect adapters to derive normalized LLM information from telemetry rows. |
| packages/app/src/llm/dashboard/LLMDashboardPage.tsx | Defines the LLM dashboard's source selection, filters, tabs, and chart composition. |
| packages/app/src/llm/dashboard/LLMSessionPanel.tsx | Queries session span summaries and expands individual entries through lazy-loaded detail views. |
| packages/app/src/llm/dashboard/SessionSpanDetail.tsx | Loads span attributes using trace, span, and timestamp identity before rendering the conversation. |
| packages/app/src/llm/components/ChatMessageItem.tsx | Renders normalized chat messages, Markdown content, and collapsible tool-call details. |
| packages/app/src/llm/lib/expressions.ts | Generates schema-aware ClickHouse expressions used by LLM dashboard aggregations. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
Telemetry[Span and log attributes] --> Normalize[LLM dialect normalization]
Normalize --> Conversation[Conversation and usage views]
Normalize --> Expressions[Read-time SQL expressions]
Expressions --> Overview[Overview charts]
Expressions --> Sessions[Session timeline]
Expressions --> Search[LLM search tables]
Sessions --> Detail[Lazy-loaded span conversation]
Reviews (4): Last reviewed commit: "feat(app): mark the LLM dashboard as bet..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| @@ -0,0 +1,316 @@ | |||
| import { useCallback, useEffect, useState } from 'react'; | |||
Contributor
There was a problem hiding this comment.
This new component is 316 lines, while LLMSessionPanel.tsx is also 302 lines. Both exceed the repository's 300-line maximum, increasing maintenance cost; split them into smaller focused components.
Context Used: AGENTS.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Comment on lines
+77
to
+81
| !isLoading && ( | ||
| <Text size="sm" c="dimmed"> | ||
| No LLM messages found on this span. Prompt and completion capture | ||
| may be disabled in the instrumentation. | ||
| </Text> |
Contributor
There was a problem hiding this comment.
This no-message branch and the no-session-results branch in LLMSessionPanel.tsx render plain Text elements instead of the required shared EmptyState, bypassing the repository's consistent empty-state presentation and behavior. Use @/components/EmptyState for both branches.
Context Used: AGENTS.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Contributor
E2E Test Results✅ All tests passed • 306 passed • 1 skipped • 1059s
Tests ran across 4 shards in parallel. |
Contributor
Deep Review✅ No critical issues found. No P0/P1 defects survive re-grading — SQL is parameterized via 🟡 P2 -- recommended
🔵 P3 nitpicks (8)
Reviewers (11): correctness, security, adversarial, performance, testing, maintainability, project-standards, kieran-typescript, frontend-races, previous-comments, agent-native. Testing gaps:
|
wrn14897
changed the title
…ion lookup hardening Post-review fixes for the LLM observability branch, driven by a live comparison against opencode's self-reported session cost: the /llm dashboard showed ~$22.72 for a session opencode itself priced at $9.16. Cost accuracy (verified exact against opencode's cost_usd on 111 calls): - Provided-cost election: apps that stamp their own per-call cost (cost_usd / llm.cost.total / gen_ai.usage.cost) are treated as the authoritative reporters, and all token/cost/call aggregations sum only those rows when any exist in scope (llmGatedSumExpr / llmGatedCountExpr, rendered as raw select aggregates). This dedupes dual-instrumented apps — opencode emits OpenInference spans (with cost) AND Vercel AI SDK spans (with gen_ai.usage.*) for every call, in separate traces, so no row-local gate could catch it. - Cache-aware estimation: the SQL cost expression now prices uncached input, cache reads (discounted), cache writes (Anthropic's 1.25x premium, new catalog rate + attribute keys incl. OpenInference prompt_details.cache_write, Vercel inputTokenDetails.cacheWriteTokens, and flat cache_creation_tokens), and output separately, matching the TS-side computeCostUsd. The inclusive/exclusive input-token heuristic now accounts for writes, and totalTokens reports effective context. - Query-size guard: the enriched cost expression embeds the price catalog per token term; the first live run exceeded ClickHouse's 256 KiB max_query_size. Rates are now factored into per-term multiIfs and the whole expression is bound once per query as a WITH expression alias (LLM_COST_SQL_ALIAS), keeping dashboard queries at ~80 KiB. Session drawer correctness (review feedback): - The per-span attribute lookup now pins TraceId alongside SpanId + timestamp (span ids can be empty or collide across traces) and is bounded to the searched window for partition pruning. Also trims the llm module's public surface to what external consumers import (fixes 23 knip unused-export findings) and replaces the unused zod chat-message schemas with plain interfaces. Adds the missing changeset for the LLM observability feature.
The LLM observability branch added 6 eslint-disable comments, tripping the app/eslint-disable ratchet (150 > baseline 144). Remove the escapes by fixing the underlying patterns instead of suppressing them: - Chat messages get a stable `id` assigned in extractConversation (conversations are immutable once extracted), so message lists key on data instead of array indexes. - Session timeline rows carry their accordion `itemValue` in the row data built per fetch, replacing the index-derived key/value pair. - The session filter now lives in the URL only: SessionSelectControlled becomes a plain value/onChange SessionSelect wired straight to the nuqs param, deleting the two deliberately-under-depped form<->URL sync effects (the drawer's "Filter dashboard" action writes the same param). - The trace-source default adoption effect gets full dependencies — the select only offers usable trace sources, so a user selection always resolves to itself and the effect can never fight it.
Adds the standard beta badge (matching Service Map's nav badge) next to the LLM breadcrumb, plus an info hover card explaining what to expect: the dashboard is experimental, which instrumentations it understands, that costs are catalog estimates unless the instrumentation reports its own cost, and how dual-instrumented apps are counted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linear
https://linear.app/clickhouse/issue/HDX-5162/llm-observability-span-chat-view-cost-tracking-sessions-and-llm
Why
Teams running LLM apps and coding agents (OpenAI/Anthropic SDKs, Vercel AI SDK, LangChain via OpenLLMetry/OpenInference, opencode, Claude Code) already send their telemetry to HyperDX — but the product had zero LLM awareness: no chat rendering, no token/cost concepts, no model analytics. This PR adds LLM observability at feature parity with dedicated tools (Langfuse dashboards, Datadog LLM Observability's operational insights) while staying HyperDX-native.
Approach: read-time, schema-agnostic
Unlike Langfuse (ingest-time workers + dedicated tables), everything here derives from span/log attribute maps at query time:
Mapand JSON-typed attribute columns via the source'seventAttributesExpressionWhat's included
Normalization lib (
packages/app/src/llm/lib) — pure, unit-tested TS:llm.input_messages, camelCaseai.usage.*, flatinput_tokens/cost_usdkeys, bracketed model ids likeclaude-opus-5[1m])Span-level UX:
model · N tokCost estimation (
lib/modelPrices.ts,lib/cost.ts):gen_ai.usage.cost,llm.cost.total,cost_usd) always wins; catalog is the fallbackmultiIfgenerator for dashboard aggregation/llmpreset dashboard (Overview | Sessions | Search), listed on the Dashboards page:gen_ai.conversation.id→session.id→ai.telemetry.metadata.sessionId) — the correlation surface for instrumentations that stamp session ids but don't propagate trace context. Row click opens a timeline drawer; each call expands into the chat viewCorrectness & performance notes
gen_ai.usage.*/llm.token_count.*/ flat primary-reporter keys) so SDK wrapper spans (e.g. Vercel'sai.streamTextarounddoStream) don't double countstopvs["stop"])Known limitations
Testing
src/llm/__tests__(per-dialect fixtures lifted from real opencode/Claude Code telemetry, cost math, SQL expression generation, tab gating, lazy-loading regression guards)DBRowSidePanel,DBTracePanel,DBRowOverviewPanel,DBTraceWaterfallChart) all green;tsc --noEmitclean; eslint 0 errors