diff --git a/public/images/docs/observe/alerts-choose-project.png b/public/images/docs/observe/alerts-choose-project.png new file mode 100644 index 00000000..53bdc1f7 Binary files /dev/null and b/public/images/docs/observe/alerts-choose-project.png differ diff --git a/public/images/docs/observe/alerts-configure.png b/public/images/docs/observe/alerts-configure.png new file mode 100644 index 00000000..9e426ac4 Binary files /dev/null and b/public/images/docs/observe/alerts-configure.png differ diff --git a/public/images/docs/observe/alerts-landing.png b/public/images/docs/observe/alerts-landing.png new file mode 100644 index 00000000..8af33fc9 Binary files /dev/null and b/public/images/docs/observe/alerts-landing.png differ diff --git a/public/images/docs/observe/alerts-select-type.png b/public/images/docs/observe/alerts-select-type.png new file mode 100644 index 00000000..7e64e128 Binary files /dev/null and b/public/images/docs/observe/alerts-select-type.png differ diff --git a/public/images/docs/observe/alerts-table.png b/public/images/docs/observe/alerts-table.png new file mode 100644 index 00000000..22e0bb6c Binary files /dev/null and b/public/images/docs/observe/alerts-table.png differ diff --git a/public/images/docs/observe/alerts-thresholds-notify.png b/public/images/docs/observe/alerts-thresholds-notify.png new file mode 100644 index 00000000..e5222f01 Binary files /dev/null and b/public/images/docs/observe/alerts-thresholds-notify.png differ diff --git a/public/images/docs/observe/dashboard-first-trace.png b/public/images/docs/observe/dashboard-first-trace.png new file mode 100644 index 00000000..076e7337 Binary files /dev/null and b/public/images/docs/observe/dashboard-first-trace.png differ diff --git a/public/images/docs/observe/dashboard-projects.png b/public/images/docs/observe/dashboard-projects.png new file mode 100644 index 00000000..b26f6d7b Binary files /dev/null and b/public/images/docs/observe/dashboard-projects.png differ diff --git a/public/images/docs/observe/dashboard-trace-detail.png b/public/images/docs/observe/dashboard-trace-detail.png new file mode 100644 index 00000000..a08b0906 Binary files /dev/null and b/public/images/docs/observe/dashboard-trace-detail.png differ diff --git a/public/images/docs/observe/dashboard-trace-table.png b/public/images/docs/observe/dashboard-trace-table.png new file mode 100644 index 00000000..af3cda47 Binary files /dev/null and b/public/images/docs/observe/dashboard-trace-table.png differ diff --git a/public/images/docs/observe/display-agent-graph.png b/public/images/docs/observe/display-agent-graph.png new file mode 100644 index 00000000..5c62a67a Binary files /dev/null and b/public/images/docs/observe/display-agent-graph.png differ diff --git a/public/images/docs/observe/display-agent-path.png b/public/images/docs/observe/display-agent-path.png new file mode 100644 index 00000000..ce682ea8 Binary files /dev/null and b/public/images/docs/observe/display-agent-path.png differ diff --git a/public/images/docs/observe/display-group-by-spans.mp4 b/public/images/docs/observe/display-group-by-spans.mp4 new file mode 100644 index 00000000..f5575647 Binary files /dev/null and b/public/images/docs/observe/display-group-by-spans.mp4 differ diff --git a/public/images/docs/observe/display-menu.png b/public/images/docs/observe/display-menu.png new file mode 100644 index 00000000..aa29a1ff Binary files /dev/null and b/public/images/docs/observe/display-menu.png differ diff --git a/public/images/docs/observe/display-reorder-columns.mp4 b/public/images/docs/observe/display-reorder-columns.mp4 new file mode 100644 index 00000000..124e8a05 Binary files /dev/null and b/public/images/docs/observe/display-reorder-columns.mp4 differ diff --git a/public/images/docs/observe/display-view-switcher.png b/public/images/docs/observe/display-view-switcher.png new file mode 100644 index 00000000..575bd915 Binary files /dev/null and b/public/images/docs/observe/display-view-switcher.png differ diff --git a/public/images/docs/observe/evals-add-button.png b/public/images/docs/observe/evals-add-button.png new file mode 100644 index 00000000..e36abfc6 Binary files /dev/null and b/public/images/docs/observe/evals-add-button.png differ diff --git a/public/images/docs/observe/evals-add-evaluation.png b/public/images/docs/observe/evals-add-evaluation.png new file mode 100644 index 00000000..555c8e1b Binary files /dev/null and b/public/images/docs/observe/evals-add-evaluation.png differ diff --git a/public/images/docs/observe/evals-configure.png b/public/images/docs/observe/evals-configure.png new file mode 100644 index 00000000..070f9b17 Binary files /dev/null and b/public/images/docs/observe/evals-configure.png differ diff --git a/public/images/docs/observe/evals-create-task.png b/public/images/docs/observe/evals-create-task.png new file mode 100644 index 00000000..c8960dcd Binary files /dev/null and b/public/images/docs/observe/evals-create-task.png differ diff --git a/public/images/docs/observe/evals-results-column.png b/public/images/docs/observe/evals-results-column.png new file mode 100644 index 00000000..d94092f4 Binary files /dev/null and b/public/images/docs/observe/evals-results-column.png differ diff --git a/public/images/docs/observe/evals-schedule.png b/public/images/docs/observe/evals-schedule.png new file mode 100644 index 00000000..d0bc4c0d Binary files /dev/null and b/public/images/docs/observe/evals-schedule.png differ diff --git a/public/images/docs/observe/evals-select-evaluation.png b/public/images/docs/observe/evals-select-evaluation.png new file mode 100644 index 00000000..4e7c46b8 Binary files /dev/null and b/public/images/docs/observe/evals-select-evaluation.png differ diff --git a/public/images/docs/observe/filters-open-panel.png b/public/images/docs/observe/filters-open-panel.png new file mode 100644 index 00000000..f80c9aaa Binary files /dev/null and b/public/images/docs/observe/filters-open-panel.png differ diff --git a/public/images/docs/observe/filters-pick-property.png b/public/images/docs/observe/filters-pick-property.png new file mode 100644 index 00000000..468dd720 Binary files /dev/null and b/public/images/docs/observe/filters-pick-property.png differ diff --git a/public/images/docs/observe/filters-pick-value.png b/public/images/docs/observe/filters-pick-value.png new file mode 100644 index 00000000..8b32730b Binary files /dev/null and b/public/images/docs/observe/filters-pick-value.png differ diff --git a/public/images/docs/observe/filters-three-modes.png b/public/images/docs/observe/filters-three-modes.png new file mode 100644 index 00000000..743a27ad Binary files /dev/null and b/public/images/docs/observe/filters-three-modes.png differ diff --git a/public/images/docs/observe/session-detail.png b/public/images/docs/observe/session-detail.png new file mode 100644 index 00000000..230039c7 Binary files /dev/null and b/public/images/docs/observe/session-detail.png differ diff --git a/public/images/docs/observe/sessions-list.png b/public/images/docs/observe/sessions-list.png new file mode 100644 index 00000000..3ca04369 Binary files /dev/null and b/public/images/docs/observe/sessions-list.png differ diff --git a/public/images/docs/observe/user-detail-sessions.png b/public/images/docs/observe/user-detail-sessions.png new file mode 100644 index 00000000..4b12119b Binary files /dev/null and b/public/images/docs/observe/user-detail-sessions.png differ diff --git a/public/images/docs/observe/user-detail-traces.png b/public/images/docs/observe/user-detail-traces.png new file mode 100644 index 00000000..abf4c7a8 Binary files /dev/null and b/public/images/docs/observe/user-detail-traces.png differ diff --git a/public/images/docs/observe/users-list.png b/public/images/docs/observe/users-list.png new file mode 100644 index 00000000..f9fed54b Binary files /dev/null and b/public/images/docs/observe/users-list.png differ diff --git a/public/images/docs/observe/views-create.png b/public/images/docs/observe/views-create.png new file mode 100644 index 00000000..4b8f6cb7 Binary files /dev/null and b/public/images/docs/observe/views-create.png differ diff --git a/public/images/docs/observe/views-save-modal.png b/public/images/docs/observe/views-save-modal.png new file mode 100644 index 00000000..980b9356 Binary files /dev/null and b/public/images/docs/observe/views-save-modal.png differ diff --git a/public/images/docs/observe/views-saved-tab.png b/public/images/docs/observe/views-saved-tab.png new file mode 100644 index 00000000..9bafcd0d Binary files /dev/null and b/public/images/docs/observe/views-saved-tab.png differ diff --git a/src/components/Sidebar.astro b/src/components/Sidebar.astro index 2df4e379..ad2ec008 100644 --- a/src/components/Sidebar.astro +++ b/src/components/Sidebar.astro @@ -10,6 +10,7 @@ import { tabNavigation, getActiveTab, getActiveGroup, type NavItem, type NavGrou const currentPath = Astro.url.pathname; const activeTab = getActiveTab(currentPath); const isDocsTab = activeTab?.tab === 'Docs'; +const isSdkTab = activeTab?.tab === 'SDK'; // For Docs tab: groups are sections you switch between // For other tabs: groups are also switchable sections @@ -21,14 +22,23 @@ let activeGroup: NavGroup | undefined; if (isDocsTab) { activeGroup = getActiveGroup(currentPath); } else { - // For non-Docs tabs, find the group containing current page - activeGroup = allGroups.find(g => - g.items.some(item => { - const norm = currentPath.replace(/\/$/, '') || '/'; - const href = (item.href || '').replace(/\/$/, '') || '/'; - return href === norm || (href !== '/' && norm.startsWith(href + '/')); - }) - ) || allGroups[0]; + // For non-Docs tabs, pick the group whose deepest matching item href is the + // longest (most specific), so a broad landing group (e.g. SDK Overview at + // /docs/sdk) never shadows a product group like Evaluation (/docs/sdk/evals). + const norm = currentPath.replace(/\/$/, '') || '/'; + let bestLen = -1; + const consider = (g: NavGroup, list: NavItem[]) => { + for (const it of list) { + if (it.href) { + const href = it.href.replace(/\/$/, '') || '/'; + const matches = href === norm || (href !== '/' && href !== '/docs' && norm.startsWith(href + '/')); + if (matches && href.length > bestLen) { bestLen = href.length; activeGroup = g; } + } + if (it.items) consider(g, it.items); + } + }; + allGroups.forEach(g => consider(g, g.items)); + if (!activeGroup) activeGroup = allGroups[0]; } const items = activeGroup?.items || []; @@ -127,6 +137,22 @@ const referenceItems: DropdownItem[] = [ { title: 'API Reference', icon: 'webhook', href: '/docs/api' }, ]; +// SDK tab: the same phase-grouped product dropdown as Docs, with the SDK's products. +const sdkPhasedOrder: { phase: string; titles: string[] }[] = [ + { phase: 'Start', titles: ['SDK Overview'] }, + { phase: 'Observe & diagnose', titles: ['traceAI'] }, + { phase: 'Evaluate & measure', titles: ['Evaluation', 'Simulation', 'Datasets'] }, + { phase: 'Improve', titles: ['Prompt Optimization', 'Annotation Queues'] }, + { phase: 'Build & connect', titles: ['Knowledge Base'] }, + { phase: 'Protect', titles: ['Protect'] }, +]; +const sdkReferenceItems: DropdownItem[] = [ + { title: 'Product Docs', icon: 'book', href: '/docs' }, + { title: 'Integrations', icon: 'plug', href: '/docs/integrations' }, + { title: 'Guides', icon: 'compass', href: '/docs/cookbook' }, + { title: 'API Reference', icon: 'webhook', href: '/docs/api' }, +]; + let dropdownSections: DropdownSection[]; if (isDocsTab) { dropdownSections = phasedOrder @@ -140,6 +166,17 @@ if (isDocsTab) { if (leftovers.length) dropdownSections.push({ phase: 'More', items: leftovers }); dropdownSections.push({ phase: 'Reference', items: referenceItems }); +} else if (isSdkTab) { + dropdownSections = sdkPhasedOrder + .map(p => ({ phase: p.phase, items: p.titles.filter(t => groupLookup.has(t)).map(toItem) })) + .filter(s => s.items.length > 0); + + // Future-proofing: any SDK group not placed above still shows under "More". + const placed = new Set(sdkPhasedOrder.flatMap(p => p.titles)); + const leftovers = allGroups.map(g => g.group).filter(t => !placed.has(t)).map(toItem); + if (leftovers.length) dropdownSections.push({ phase: 'More', items: leftovers }); + + dropdownSections.push({ phase: 'Reference', items: sdkReferenceItems }); } else { // Other multi-group tabs: one unlabelled section (empty phase hides the header). dropdownSections = [{ phase: '', items: allGroups.map(g => toItem(g.group)) }]; diff --git a/src/lib/navigation.ts b/src/lib/navigation.ts index 6bd49aac..df36dfb7 100644 --- a/src/lib/navigation.ts +++ b/src/lib/navigation.ts @@ -368,118 +368,49 @@ export const tabNavigation: NavTab[] = [ group: 'Observability', icon: 'eye', items: [ - { title: 'Get Started', href: '/docs/observe' }, + { title: 'Overview', href: '/docs/observe' }, { title: 'Quickstart', href: '/docs/observe/quickstart' }, { title: 'Concepts', items: [ - { title: 'Understanding Observability', href: '/docs/tracing/concepts' }, { title: 'Spans', href: '/docs/observe/concepts/spans' }, { title: 'Traces', href: '/docs/observe/concepts/traces' }, - { title: 'OpenTelemetry', href: '/docs/tracing/concepts/otel' }, - { title: 'traceAI', href: '/docs/tracing/concepts/traceai' }, + { title: 'Sessions', href: '/docs/observe/concepts/sessions' }, + { title: 'Users', href: '/docs/observe/concepts/users' }, + { title: 'Voice observability', href: '/docs/observe/concepts/voice-observability' }, + { title: 'Observability model', href: '/docs/observe/concepts/observability-model' }, ] }, { - title: 'Features', + title: 'Guides', items: [ - { title: 'Run Evals on Traces', href: '/docs/observe/features/evals' }, - { title: 'Sessions', href: '/docs/observe/features/session' }, - { title: 'Users', href: '/docs/observe/features/users' }, - { title: 'Alerts & Monitors', href: '/docs/observe/features/alerts' }, - { title: 'Voice Observability', href: '/docs/observe/features/voice' }, - { title: 'Dashboards', href: '/docs/observe/features/dashboard' }, { - title: 'Manual Tracing', + title: 'Explore dashboard', items: [ - { title: 'Set Up Tracing', href: '/docs/observe/features/manual-tracing/set-up-tracing' }, - { title: 'Instrument with traceAI Helpers', href: '/docs/observe/features/manual-tracing/instrument-with-traceai-helpers' }, - { title: 'Get Current Tracer and Span', href: '/docs/observe/features/manual-tracing/get-current-span-context' }, - { title: 'Enriching Spans with Attributes, Metadata, and Tags', href: '/docs/observe/features/manual-tracing/add-attributes-metadata-tags' }, - { title: 'Logging Prompt Templates & Variables', href: '/docs/observe/features/manual-tracing/log-prompt-templates' }, - { title: 'Events, Exceptions, and Status', href: '/docs/observe/features/manual-tracing/add-events-exceptions-status' }, - { title: 'Set Session ID and User ID', href: '/docs/observe/features/manual-tracing/set-session-user-id' }, - { title: 'Tool Spans Creation', href: '/docs/observe/features/manual-tracing/create-tool-spans' }, - { title: 'Mask Span Attributes', href: '/docs/observe/features/manual-tracing/mask-span-attributes' }, - { title: 'Advanced Tracing (OTEL)', href: '/docs/observe/features/manual-tracing/advanced-tracing-examples' }, - { title: 'FI Semantic Conventions', href: '/docs/observe/features/manual-tracing/semantic-conventions' }, - { title: 'In-line Evaluations', href: '/docs/observe/features/manual-tracing/in-line-evals' }, - { title: 'Adding Annotations to your Spans', href: '/docs/observe/features/manual-tracing/annotating-using-api' }, - { title: 'Langfuse Integration', href: '/docs/observe/features/manual-tracing/langfuse-integration' }, + { title: 'Overview', href: '/docs/observe/guides/explore-dashboard' }, + { title: 'Filters', href: '/docs/observe/guides/explore-dashboard/filters' }, + { title: 'Views', href: '/docs/observe/guides/explore-dashboard/views' }, + { title: 'Display options', href: '/docs/observe/guides/explore-dashboard/display-options' }, ] }, + { title: 'Setup alerts', href: '/docs/observe/guides/setup-alerts' }, + { title: 'Setup evals', href: '/docs/observe/guides/setup-evals' }, ] }, { - title: 'Integration', + title: 'Reference', items: [ - { title: 'Overview', href: '/docs/tracing/auto' }, - { - title: 'LLM Providers', - items: [ - { title: 'OpenAI', href: '/docs/tracing/auto/openai' }, - { title: 'Anthropic', href: '/docs/tracing/auto/anthropic' }, - { title: 'AWS Bedrock', href: '/docs/tracing/auto/bedrock' }, - { title: 'Vertex AI', href: '/docs/tracing/auto/vertexai' }, - { title: 'Google GenAI', href: '/docs/tracing/auto/google_genai' }, - { title: 'Google ADK', href: '/docs/tracing/auto/google_adk' }, - { title: 'Groq', href: '/docs/tracing/auto/groq' }, - { title: 'MistralAI', href: '/docs/tracing/auto/mistralai' }, - { title: 'Together AI', href: '/docs/tracing/auto/togetherai' }, - { title: 'Ollama', href: '/docs/tracing/auto/ollama' }, - { title: 'Portkey', href: '/docs/tracing/auto/portkey' }, - ] - }, - { - title: 'Frameworks & Agents', - items: [ - { title: 'LangChain', href: '/docs/tracing/auto/langchain' }, - { title: 'LangGraph', href: '/docs/tracing/auto/langgraph' }, - { title: 'LlamaIndex', href: '/docs/tracing/auto/llamaindex' }, - { title: 'LlamaIndex Workflows', href: '/docs/tracing/auto/llamaindex-workflows' }, - { title: 'LiteLLM', href: '/docs/tracing/auto/litellm' }, - { title: 'CrewAI', href: '/docs/tracing/auto/crewai' }, - { title: 'AutoGen', href: '/docs/tracing/auto/autogen' }, - { title: 'Haystack', href: '/docs/tracing/auto/haystack' }, - { title: 'DSPy', href: '/docs/tracing/auto/dspy' }, - { title: 'OpenAI Agents', href: '/docs/tracing/auto/openai_agents' }, - { title: 'Smol Agents', href: '/docs/tracing/auto/smol_agents' }, - { title: 'Instructor', href: '/docs/tracing/auto/instructor' }, - { title: 'PromptFlow', href: '/docs/tracing/auto/promptflow' }, - { title: 'Guardrails', href: '/docs/tracing/auto/guardrails' }, - { title: 'MCP', href: '/docs/tracing/auto/mcp' }, - { title: 'Mastra', href: '/docs/tracing/auto/mastra' }, - { title: 'Vercel AI SDK', href: '/docs/tracing/auto/vercel' }, - ] - }, - { - title: 'Voice & Realtime', - items: [ - { title: 'LiveKit', href: '/docs/tracing/auto/livekit' }, - { title: 'Pipecat', href: '/docs/tracing/auto/pipecat' }, - ] - }, - { - title: 'Java', - items: [ - { title: 'Overview', href: '/docs/tracing/auto/java' }, - { title: 'Spring Boot', href: '/docs/tracing/auto/spring-boot' }, - { title: 'OpenAI', href: '/docs/tracing/auto/java/openai' }, - { title: 'Anthropic', href: '/docs/tracing/auto/java/anthropic' }, - { title: 'AWS Bedrock', href: '/docs/tracing/auto/java/bedrock' }, - { title: 'Cohere', href: '/docs/tracing/auto/java/cohere' }, - { title: 'Pinecone', href: '/docs/tracing/auto/java/pinecone' }, - { title: 'LLM Providers', href: '/docs/tracing/auto/java/llm-providers' }, - { title: 'Vector Databases', href: '/docs/tracing/auto/java/vector-databases' }, - { title: 'Frameworks', href: '/docs/tracing/auto/java/frameworks' }, - ] - }, - { - title: 'Other', - items: [ - { title: 'n8n', href: '/docs/integrations/traceai/n8n' }, - ] - }, + { title: 'Filters', href: '/docs/observe/reference/filters' }, + { title: 'traceAI', href: '/docs/observe/concepts/traceai' }, + ] + }, + { + title: 'Troubleshooting', + items: [ + { title: 'No traces appear', href: '/docs/observe/troubleshooting/no-traces-appearing' }, + { title: 'Missing spans or fields', href: '/docs/observe/troubleshooting/missing-attributes' }, + { title: 'Dashboard numbers look wrong', href: '/docs/observe/troubleshooting/dashboard-numbers-look-wrong' }, + { title: 'Alerts not firing', href: '/docs/observe/troubleshooting/alerts-did-not-fire' }, ] }, ] @@ -907,7 +838,7 @@ export const tabNavigation: NavTab[] = [ title: 'Observability', icon: 'eye', items: [ - { title: 'Implement Observability', href: '/docs/cookbook/observability' }, + { title: 'Observing a LangGraph agent and obtaining insights', href: '/docs/cookbook/observe-langgraph-agent-and-obtain-insights' }, { title: 'Text-to-SQL Evaluation', href: '/docs/cookbook/text-to-sql' }, ] }, @@ -958,50 +889,142 @@ export const tabNavigation: NavTab[] = [ href: '/docs/sdk', groups: [ { - group: 'SDK Reference', + group: 'SDK Overview', + icon: 'code', items: [ { title: 'SDK Overview', href: '/docs/sdk' }, { - title: 'AI Evaluation', + title: 'List of SDKs', items: [ - { title: 'Overview', href: '/docs/sdk/evals' }, - { title: 'Running Evaluations', href: '/docs/sdk/evals/evaluate' }, - { title: 'Distributed Evaluator', href: '/docs/sdk/evals/distributed' }, - { title: 'AutoEval', href: '/docs/sdk/evals/autoeval' }, - { title: 'Guardrails', href: '/docs/sdk/evals/guardrails-module' }, - { title: 'Local & Hybrid', href: '/docs/sdk/evals/local' }, - { title: 'OpenTelemetry', href: '/docs/sdk/evals/otel' }, - { title: 'Code Security', href: '/docs/sdk/evals/code-security' }, - { - title: 'Metrics Reference', - items: [ - { title: 'Overview', href: '/docs/sdk/evals/metrics' }, - { title: 'String & Similarity', href: '/docs/sdk/evals/metrics/string' }, - { title: 'JSON & Structured', href: '/docs/sdk/evals/metrics/json' }, - { title: 'Hallucination', href: '/docs/sdk/evals/metrics/hallucination' }, - { title: 'RAG', href: '/docs/sdk/evals/metrics/rag' }, - { title: 'Agents & Functions', href: '/docs/sdk/evals/metrics/agents' }, - { title: 'Guardrails', href: '/docs/sdk/evals/metrics/guardrails' }, - ] - }, - { title: 'Cloud Evals', href: '/docs/sdk/evals/cloud-evals' }, - { title: 'LLM-as-Judge', href: '/docs/sdk/evals/llm-judge' }, - { title: 'Streaming', href: '/docs/sdk/evals/streaming' }, - { title: 'Feedback Loops', href: '/docs/sdk/evals/feedback' }, + { title: 'Evaluation', href: '/docs/sdk/list/evaluation' }, + { title: 'TraceAI', href: '/docs/sdk/list/traceai' }, + { title: 'Core SDKs', href: '/docs/sdk/list/core' }, ] }, + ] + }, + { + group: 'Evaluation', + icon: 'chart', + items: [ + { title: 'Overview', href: '/docs/sdk/evals' }, + { title: 'Running Evaluations', href: '/docs/sdk/evals/evaluate' }, + { title: 'AutoEval', href: '/docs/sdk/evals/autoeval' }, + { title: 'LLM-as-Judge', href: '/docs/sdk/evals/llm-judge' }, + { title: 'Guardrails', href: '/docs/sdk/evals/guardrails-module' }, + { title: 'Local & Hybrid', href: '/docs/sdk/evals/local' }, + { title: 'Distributed Evaluator', href: '/docs/sdk/evals/distributed' }, + { title: 'Streaming', href: '/docs/sdk/evals/streaming' }, + { title: 'Cloud Evals', href: '/docs/sdk/evals/cloud-evals' }, + { title: 'Feedback Loops', href: '/docs/sdk/evals/feedback' }, + { title: 'Code Security', href: '/docs/sdk/evals/code-security' }, + { title: 'OpenTelemetry', href: '/docs/sdk/evals/otel' }, + { + title: 'Metrics Reference', + items: [ + { title: 'Overview', href: '/docs/sdk/evals/metrics' }, + { title: 'String & Similarity', href: '/docs/sdk/evals/metrics/string' }, + { title: 'JSON & Structured', href: '/docs/sdk/evals/metrics/json' }, + { title: 'Hallucination', href: '/docs/sdk/evals/metrics/hallucination' }, + { title: 'RAG', href: '/docs/sdk/evals/metrics/rag' }, + { title: 'Agents & Functions', href: '/docs/sdk/evals/metrics/agents' }, + { title: 'Guardrails', href: '/docs/sdk/evals/metrics/guardrails' }, + ] + }, + ] + }, + { + group: 'traceAI', + icon: 'eye', + items: [ + { title: 'Overview', href: '/docs/sdk/tracing' }, { - title: 'Core SDK', + title: 'How-to guides', items: [ - { title: 'Datasets', href: '/docs/sdk/datasets' }, - { title: 'Tracing', href: '/docs/sdk/tracing' }, - { title: 'Protect', href: '/docs/sdk/protect' }, - { title: 'Knowledge Base', href: '/docs/sdk/knowledgebase' }, - { title: 'Annotation Queues', href: '/docs/sdk/annotation-queues' }, - { title: 'Prompt Optimization', href: '/docs/sdk/optimization' }, - { title: 'Simulation Testing', href: '/docs/sdk/simulate' }, + { title: 'Set up tracing', href: '/docs/sdk/tracing/set-up-tracing' }, + { title: 'Instrument with helpers', href: '/docs/sdk/tracing/instrument-with-traceai-helpers' }, + { title: 'Set session & user IDs', href: '/docs/sdk/tracing/set-session-user-id' }, + { title: 'Attributes, metadata & tags', href: '/docs/sdk/tracing/add-attributes-metadata-tags' }, + { title: 'Log prompt templates', href: '/docs/sdk/tracing/log-prompt-templates' }, + { title: 'Events, exceptions & status', href: '/docs/sdk/tracing/add-events-exceptions-status' }, + { title: 'Mask attributes', href: '/docs/sdk/tracing/mask-span-attributes' }, + { title: 'Create tool spans', href: '/docs/sdk/tracing/create-tool-spans' }, + { title: 'Get span context', href: '/docs/sdk/tracing/get-current-span-context' }, + { title: 'In-line evals', href: '/docs/sdk/tracing/in-line-evals' }, + { title: 'Annotate via API', href: '/docs/sdk/tracing/annotating-using-api' }, + { title: 'Advanced examples', href: '/docs/sdk/tracing/advanced-tracing-examples' }, + { title: 'Langfuse integration', href: '/docs/sdk/tracing/langfuse-integration' }, ] }, + { + title: 'Reference', + items: [ + { title: 'register()', href: '/docs/sdk/tracing/register' }, + { title: 'FITracer & custom spans', href: '/docs/sdk/tracing/fitracer' }, + { title: 'Context helpers', href: '/docs/sdk/tracing/context-helpers' }, + { title: 'TraceConfig', href: '/docs/sdk/tracing/trace-config' }, + { title: 'EvalTags', href: '/docs/sdk/tracing/eval-tags' }, + { title: 'Instrumentors', href: '/docs/sdk/tracing/instrumentors' }, + { title: 'Environment variables', href: '/docs/sdk/tracing/environment-variables' }, + { title: 'Semantic conventions', href: '/docs/sdk/tracing/semantic-conventions' }, + ] + }, + ] + }, + { + group: 'Simulation', + icon: 'play', + items: [ + { title: 'Overview', href: '/docs/sdk/simulate' }, + ] + }, + { + group: 'Datasets', + icon: 'table', + items: [ + { title: 'Overview', href: '/docs/sdk/datasets' }, + ] + }, + { + group: 'Prompt Optimization', + icon: 'gauge', + items: [ + { title: 'Overview', href: '/docs/sdk/optimization' }, + ] + }, + { + group: 'Annotation Queues', + icon: 'pen', + items: [ + { title: 'Overview', href: '/docs/sdk/annotation-queues' }, + { + title: 'Concepts', + items: [ + { title: 'Labels', href: '/docs/sdk/annotation-queues/labels' }, + { title: 'Queue management', href: '/docs/sdk/annotation-queues/queues' }, + { title: 'Queue lifecycle', href: '/docs/sdk/annotation-queues/lifecycle' }, + { title: 'Queue items', href: '/docs/sdk/annotation-queues/items' }, + { title: 'Annotations', href: '/docs/sdk/annotation-queues/annotations' }, + { title: 'Scores', href: '/docs/sdk/annotation-queues/scores' }, + { title: 'Progress & analytics', href: '/docs/sdk/annotation-queues/analytics' }, + { title: 'Export', href: '/docs/sdk/annotation-queues/export' }, + { title: 'Data models', href: '/docs/sdk/annotation-queues/data-models' }, + ] + }, + ] + }, + { + group: 'Knowledge Base', + icon: 'brain', + items: [ + { title: 'Overview', href: '/docs/sdk/knowledgebase' }, + ] + }, + { + group: 'Protect', + icon: 'shield', + items: [ + { title: 'Overview', href: '/docs/sdk/protect' }, ] } ] diff --git a/src/lib/redirects.ts b/src/lib/redirects.ts index 2625e00c..1e9cd2ed 100644 --- a/src/lib/redirects.ts +++ b/src/lib/redirects.ts @@ -1,7 +1,27 @@ // Auto-generated redirect map: old Mintlify URLs โ†’ new docs URLs // 275 redirects from futureagi.mintlify.app export const redirectMap: Record = { + // Manual-instrumentation pages moved from Observe features into the traceAI SDK section + '/docs/observe/features/manual-tracing/set-up-tracing': '/docs/sdk/tracing/set-up-tracing', + '/docs/observe/features/manual-tracing/instrument-with-traceai-helpers': '/docs/sdk/tracing/instrument-with-traceai-helpers', + '/docs/observe/features/manual-tracing/set-session-user-id': '/docs/sdk/tracing/set-session-user-id', + '/docs/observe/features/manual-tracing/add-attributes-metadata-tags': '/docs/sdk/tracing/add-attributes-metadata-tags', + '/docs/observe/features/manual-tracing/log-prompt-templates': '/docs/sdk/tracing/log-prompt-templates', + '/docs/observe/features/manual-tracing/add-events-exceptions-status': '/docs/sdk/tracing/add-events-exceptions-status', + '/docs/observe/features/manual-tracing/mask-span-attributes': '/docs/sdk/tracing/mask-span-attributes', + '/docs/observe/features/manual-tracing/create-tool-spans': '/docs/sdk/tracing/create-tool-spans', + '/docs/observe/features/manual-tracing/get-current-span-context': '/docs/sdk/tracing/get-current-span-context', + '/docs/observe/features/manual-tracing/in-line-evals': '/docs/sdk/tracing/in-line-evals', + '/docs/observe/features/manual-tracing/annotating-using-api': '/docs/sdk/tracing/annotating-using-api', + '/docs/observe/features/manual-tracing/semantic-conventions': '/docs/sdk/tracing/semantic-conventions', + '/docs/observe/features/manual-tracing/advanced-tracing-examples': '/docs/sdk/tracing/advanced-tracing-examples', + '/docs/observe/features/manual-tracing/langfuse-integration': '/docs/sdk/tracing/langfuse-integration', + '/docs/cookbook/observability': '/docs/cookbook/observe-langgraph-agent-and-obtain-insights', + '/docs/cookbook/improve-langgraph-agent-with-observability': '/docs/cookbook/observe-langgraph-agent-and-obtain-insights', '/docs/observe/features/annotation-queue-using-sdk': '/docs/annotations/sdk/annotation-queue-using-sdk', + // SDK pages restructured: sdk/tracing.mdx flat page โ†’ sdk/tracing/ folder (index still serves /docs/sdk/tracing); annotation-queues moved under Annotations + '/docs/sdk/tracing': '/docs/sdk/tracing/set-up-tracing', + '/docs/sdk/annotation-queues': '/docs/annotations/sdk/annotation-queue-using-sdk', '/docs/observe/voice/set-up': '/docs/observe/features/voice', '/docs/quickstart/installation': '/docs/installation', '/docs/observability': '/docs/tracing/auto', @@ -14,17 +34,17 @@ export const redirectMap: Record = { '/docs/knowledge-base/concept': '/docs/knowledge-base/concepts/concept', '/docs/prompt-workbench': '/docs/prompt', '/docs/prompt-workbench/sdk': '/docs/prompt/features/sdk', - '/docs/tracing/manual/log-prompt-templates': '/docs/observe/features/manual-tracing/log-prompt-templates', - '/docs/tracing/manual/in-line-evals': '/docs/observe/features/manual-tracing/in-line-evals', + '/docs/tracing/manual/log-prompt-templates': '/docs/sdk/tracing/log-prompt-templates', + '/docs/tracing/manual/in-line-evals': '/docs/sdk/tracing/in-line-evals', '/docs/simulation/set-up/scenarios': '/docs/simulation/concepts/scenarios', '/docs/simulation/set-up/agent-definition': '/docs/simulation/concepts/agent-definition', '/docs/tracing/concepts/components': '/docs/tracing/concepts', - '/docs/tracing/manual/add-attributes-metadata-tags': '/docs/observe/features/manual-tracing/add-attributes-metadata-tags', - '/docs/tracing/manual/add-events-exceptions-status': '/docs/observe/features/manual-tracing/add-events-exceptions-status', - '/docs/tracing/manual/advanced-tracing-examples': '/docs/observe/features/manual-tracing/advanced-tracing-examples', - '/docs/tracing/manual/langfuse-integration': '/docs/observe/features/manual-tracing/langfuse-integration', - '/docs/tracing/manual/set-session-user-id': '/docs/observe/features/manual-tracing/set-session-user-id', - '/docs/tracing/manual/set-up-tracing': '/docs/observe/features/manual-tracing/set-up-tracing', + '/docs/tracing/manual/add-attributes-metadata-tags': '/docs/sdk/tracing/add-attributes-metadata-tags', + '/docs/tracing/manual/add-events-exceptions-status': '/docs/sdk/tracing/add-events-exceptions-status', + '/docs/tracing/manual/advanced-tracing-examples': '/docs/sdk/tracing/advanced-tracing-examples', + '/docs/tracing/manual/langfuse-integration': '/docs/sdk/tracing/langfuse-integration', + '/docs/tracing/manual/set-session-user-id': '/docs/sdk/tracing/set-session-user-id', + '/docs/tracing/manual/set-up-tracing': '/docs/sdk/tracing/set-up-tracing', '/docs/prism': '/docs/command-center', '/docs/prism/concepts/api-reference': '/docs/command-center/concepts/api-reference', '/docs/prism/concepts/configuration': '/docs/command-center/concepts/configuration', @@ -77,7 +97,7 @@ export const redirectMap: Record = { '/cookbook/cookbook5/How-to-build-and-incrementally-improve-RAG-applications-in-Langchain': '/docs/cookbook/rag-langchain', '/cookbook/cookbook6/How-to-evaluate-RAG-Applications': '/docs/cookbook/evaluate-rag', '/cookbook/cookbook7/Creating-Trustworthy-RAGs-for-Chatbots': '/docs/cookbook/trustworthy-rag', - '/cookbook/cookbook8/How-To-Implement-Observability': '/docs/cookbook/observability', + '/cookbook/cookbook8/How-To-Implement-Observability': '/docs/cookbook/observe-langgraph-agent-and-obtain-insights', '/cookbook/cookbook9/How-To-Decrease-RAG-Hallucination': '/docs/cookbook/decrease-hallucination', '/cookbook/integrations/mongodb': '/docs/cookbook/mongodb', '/cookbook/optimization/basic-prompt-optimization': '/docs/cookbook/basic-optimization', @@ -163,20 +183,20 @@ export const redirectMap: Record = { '/future-agi/get-started/knowledge-base/how-to/create-kb-using-sdk': '/docs/knowledge-base/features/sdk', '/future-agi/get-started/knowledge-base/how-to/create-kb-using-ui': '/docs/knowledge-base/features/ui', '/future-agi/get-started/knowledge-base/overview': '/docs/knowledge-base', - '/future-agi/get-started/observability/manual-tracing/add-attributes-metadata-tags': '/docs/observe/features/manual-tracing/add-attributes-metadata-tags', - '/future-agi/get-started/observability/manual-tracing/add-events-exceptions-status': '/docs/observe/features/manual-tracing/add-events-exceptions-status', - '/future-agi/get-started/observability/manual-tracing/advanced-tracing-examples': '/docs/observe/features/manual-tracing/advanced-tracing-examples', - '/future-agi/get-started/observability/manual-tracing/annotating-using-api': '/docs/observe/features/manual-tracing/annotating-using-api', - '/future-agi/get-started/observability/manual-tracing/create-tool-spans': '/docs/observe/features/manual-tracing/create-tool-spans', - '/future-agi/get-started/observability/manual-tracing/get-current-span-context': '/docs/observe/features/manual-tracing/get-current-span-context', - '/future-agi/get-started/observability/manual-tracing/in-line-evals': '/docs/observe/features/manual-tracing/in-line-evals', - '/future-agi/get-started/observability/manual-tracing/instrument-with-traceai-helpers': '/docs/observe/features/manual-tracing/instrument-with-traceai-helpers', - '/future-agi/get-started/observability/manual-tracing/langfuse-intergation': '/docs/observe/features/manual-tracing/langfuse-integration', - '/future-agi/get-started/observability/manual-tracing/log-prompt-templates': '/docs/observe/features/manual-tracing/log-prompt-templates', - '/future-agi/get-started/observability/manual-tracing/mask-span-attributes': '/docs/observe/features/manual-tracing/mask-span-attributes', - '/future-agi/get-started/observability/manual-tracing/semantic-conventions': '/docs/observe/features/manual-tracing/semantic-conventions', - '/future-agi/get-started/observability/manual-tracing/set-session-user-id': '/docs/observe/features/manual-tracing/set-session-user-id', - '/future-agi/get-started/observability/manual-tracing/set-up-tracing': '/docs/observe/features/manual-tracing/set-up-tracing', + '/future-agi/get-started/observability/manual-tracing/add-attributes-metadata-tags': '/docs/sdk/tracing/add-attributes-metadata-tags', + '/future-agi/get-started/observability/manual-tracing/add-events-exceptions-status': '/docs/sdk/tracing/add-events-exceptions-status', + '/future-agi/get-started/observability/manual-tracing/advanced-tracing-examples': '/docs/sdk/tracing/advanced-tracing-examples', + '/future-agi/get-started/observability/manual-tracing/annotating-using-api': '/docs/sdk/tracing/annotating-using-api', + '/future-agi/get-started/observability/manual-tracing/create-tool-spans': '/docs/sdk/tracing/create-tool-spans', + '/future-agi/get-started/observability/manual-tracing/get-current-span-context': '/docs/sdk/tracing/get-current-span-context', + '/future-agi/get-started/observability/manual-tracing/in-line-evals': '/docs/sdk/tracing/in-line-evals', + '/future-agi/get-started/observability/manual-tracing/instrument-with-traceai-helpers': '/docs/sdk/tracing/instrument-with-traceai-helpers', + '/future-agi/get-started/observability/manual-tracing/langfuse-intergation': '/docs/sdk/tracing/langfuse-integration', + '/future-agi/get-started/observability/manual-tracing/log-prompt-templates': '/docs/sdk/tracing/log-prompt-templates', + '/future-agi/get-started/observability/manual-tracing/mask-span-attributes': '/docs/sdk/tracing/mask-span-attributes', + '/future-agi/get-started/observability/manual-tracing/semantic-conventions': '/docs/sdk/tracing/semantic-conventions', + '/future-agi/get-started/observability/manual-tracing/set-session-user-id': '/docs/sdk/tracing/set-session-user-id', + '/future-agi/get-started/observability/manual-tracing/set-up-tracing': '/docs/sdk/tracing/set-up-tracing', '/future-agi/get-started/optimization/dataset-optimization': '/docs/cookbook/quickstart/dataset-optimization', '/future-agi/get-started/optimization/how-to/using-python-sdk': '/docs/optimization/features/using-python-sdk', '/future-agi/get-started/optimization/optimizers/bayesian-search': '/docs/optimization/optimizers/bayesian-search', diff --git a/src/pages/docs/cookbook/index.mdx b/src/pages/docs/cookbook/index.mdx index ba001f1b..10500ec3 100644 --- a/src/pages/docs/cookbook/index.mdx +++ b/src/pages/docs/cookbook/index.mdx @@ -111,11 +111,11 @@ description: "Practical step-by-step guides for evaluation, optimization, simula - Add monitoring and observability to your AI applications + Trace a support agent by session and user, score it with an Eval Task, and read the scores as insights Please export your OpenAI and FutureAGI api keys before proceeding to run the code -### 1. Basic Setup - -```python - -# export FI_API_KEY="xxxasxas" -# export FI_SECRET_KEY="hasdaxxasa21" -# export OPENAI_API_KEY="jasfapsd" - -import os -import gradio as gr - -from langchain_openai import ChatOpenAI -from fi_instrumentation import register -from fi_instrumentation.fi_types import ( - EvalName, - EvalSpanKind, - EvalTag, - EvalTagType, - ProjectType -) - -# Initialize tracing -trace_provider = register( -project_type=ProjectType.OBSERVE, - project_name="Your-Project-Name" -) - -``` - -## Real-World Application Example - -Let's consider a simplified example of a chat application that uses observability. This example illustrates a chatbot application that has Observability in place. - -### Application Overview - -This Gradio-based chat app includes: - -- Integration of OpenAI's GPT model -- Monitoring of real-time responses -- Easy-to-use interface -- Full observability metrics - -### Code Implementation - -```python -import os -import gradio as gr - -from langchain_openai import ChatOpenAI -from fi_instrumentation import register -from traceai_langchain import LangChainInstrumentor -from fi_instrumentation.fi_types import ( - EvalName, - EvalSpanKind, - EvalTag, - EvalTagType, - ProjectType -) - -# Set up tracing with FutureAGI -trace_provider = register( - project_type=ProjectType.OBSERVE, - project_name="Simple-Chat-App" -) - -LangChainInstrumentor().instrument(tracer_provider=trace_provider) - -# Set up the LLM -llm = ChatOpenAI(temperature=0, model="gpt-4o-mini") - -def process_message(message, history): - """Process user message and generate response with observability""" - try: - # Generate response using LLM - response = llm.invoke(message) - - # Return formatted response - return history + [(message, response.content)] - except Exception as e: - error_message = f"Sorry, I encountered an error: {str(e)}" - return history + [(message, error_message)] - -def main(): - with gr.Blocks(theme=gr.themes.Soft()) as demo: - # Create chat interface - chatbot = gr.Chatbot( - label="Simple Chat Assistant", - height=400, - value=[], - type="chat", - autoscroll=True - ) - - with gr.Row(): - msg = gr.Textbox( - label="Message", - placeholder="Type your message here.", - scale=4, - container=False, - autofocus=True, - show_label=False - ) - submit_button = gr.Button( - "Send", - variant="primary", - scale=1, - size="sm" - ) - - # Example queries - gr.Examples( - examples=[ - "What is artificial intelligence?", - "Describe quantum computing in everyday language", - "What are the advantages of observability?", - ], - inputs=msg - ) - - # Handle message submission - msg.submit( - fn=process_message, - inputs=[msg, chatbot], - outputs=[chatbot], - queue=False - ).then( - lambda: "", - None, - msg, - queue=False - ) - - # Also trigger on button click - submit_button.click( - fn=process_message, - inputs=[msg, chatbot], - outputs=[chatbot], - queue=False - ).then( - lambda: "", - None, - msg, - queue=False - ) - - # Launch the demo - demo.launch( - share=True, - show_error=True - ) - -if __name__ == "__main__": - main() -``` - -After this application is installed we can then monitor and configure different features offered by FutureAGI in the dashboard. We can create an Eval Task to evaluate our data generated by the app. - -![FutureAGI Dashboard](/images/docs/cookbook-observability/c81.png)
Dashboard from FutureAGI platform showcasing our deployed application in OBSERVE.
- -To check a specific event for a trace of an application, we can click on one of the traces and check out the flow of our application and its individual events (spans). -![FutureAGI Trace](/images/docs/cookbook-observability/c82.png)
Trace Tree that shows the detailed overview of application session
-### Key Features Explained - -1. **Observability Setup** - - Integration of FutureAGI's instrumentation framework - - Monitoring response quality - - Tracking automatic LLM interaction -2. **Gradio Interface** - - Responsive, modern design - - Live chat functionality - - Integrated error handling -- Example queries for testing -3. **Monitoring Capabilities** - - Response quality metrics - - Error rate monitoring - - Performance monitoring - -## Best Practices for Implementation - -1. **Performance Optimization** - - Employ suitable sampling rates - - Instrumentation overhead monitoring - - Cache strategies implementation -2. **Error Handling** - - Comprehensive error logging -- Friendly error messages -- Gracious degradation -3. **Security Considerations** - - Secure API credentials - - Protection of data privacy - - Implementing access control - -## Common Challenges and Solutions - -| Challenge | Solution | Impact | -| --- | --- | --- | -| High Overhead | Adopt sampling | Lowered resource consumption -| Data Privacy | Utilize data masking | Secure user data | -| Complexity | Utilize auto-instrumentation setup | Simplified implementation | - -## FAQs - -### 1. What is the lowest supported Python version? - -Python 3.10 or later is recommended for best compatibility with FutureAGI's instrumentation framework. - -### 2. How does observability affect application performance? - -The impact on performance becomes negligible when properly used (usually <1% overhead), providing immense value in terms of insights. - -### 3. Can I add observability to current applications? - -Yes, observability can be incorporated into current applications with limited code modification. - -### 4. What kind of metrics can I monitor? - -You can monitor various metrics such as: - -- Latency -- Error rates -- Resource consumption -- Tokens Used -- Cost of workflow -- Evaluation Metrics - -## Next Steps - -Ready to add observability to your app? Here are the steps: - -1. Create an account on FutureAGI -2. Install the necessary packages -3. Add basic instrumentation -4. Monitor and optimize - -## Additional Resources - -- [FutureAGI Documentation](https://docs.futureagi.com/) -- [Gradio Documentation](https://gradio.app/docs) - -Begin implementing observability in your Python AI applications today! Sign up for a free FutureAGI account and start monitoring your application's performance and reliability. - -๐Ÿ“ฉ Subscribe to our [newsletter](https://futureagi.com/blogs) for weekly AI development tips and best practices! \ No newline at end of file diff --git a/src/pages/docs/cookbook/observe-langgraph-agent-and-obtain-insights.mdx b/src/pages/docs/cookbook/observe-langgraph-agent-and-obtain-insights.mdx new file mode 100644 index 00000000..407d6c32 --- /dev/null +++ b/src/pages/docs/cookbook/observe-langgraph-agent-and-obtain-insights.mdx @@ -0,0 +1,268 @@ +--- +title: "Observing a LangGraph agent and obtaining insights" +description: "Instrument a LangGraph support agent with Future AGI, group its traces by session and user, score every turn with an Observe Eval Task, and read the scores to find which layer fails and for whom." +--- + + +Build a multi-turn LangGraph support agent and instrument it with [traceAI](/docs/observe/concepts/traceai). Every conversation lands in [Observe](/docs/observe), grouped by [session](/docs/observe/concepts/sessions) and [user](/docs/observe/concepts/users). Then read three things back: the traces of what the agent did, an [Eval Task](/docs/observe/guides/setup-evals) that scores every turn, and filters that turn those scores into insight โ€” which layer failed, on which turns, for which customer. + + +| Time | Difficulty | +|------|------------| +| 25-35 min | Intermediate | + +The loop you'll run: + + B["Read the traces"] + B --> C["Score with an Eval Task"] + C --> D["Turn scores into insight"] + D --> E["Alert & save a view"]`} /> + + +- FutureAGI account โ†’ [app.futureagi.com](https://app.futureagi.com) +- API keys: `FI_API_KEY` and `FI_SECRET_KEY` (see [Get your API keys](/docs/admin-settings)) +- An `OPENAI_API_KEY` +- Python 3.11 + + +## Install + +```bash +pip install fi-instrumentation-otel traceAI-langchain langgraph langchain-openai langchain-core +``` + +```bash +export FI_API_KEY="your-api-key" +export FI_SECRET_KEY="your-secret-key" +export OPENAI_API_KEY="your-openai-api-key" +``` + +## Tutorial + + + + + +Prove your keys and ingestion work before writing any agent code. Register against an **Observe** project, emit one span, and flush. + +```python +from fi_instrumentation import register, FITracer +from fi_instrumentation.fi_types import ProjectType + +# Reuse this trace_provider and tracer for the rest of the cookbook. +trace_provider = register( + project_type=ProjectType.OBSERVE, + project_name="support-agent", + set_global_tracer_provider=True, +) +tracer = FITracer(trace_provider.get_tracer(__name__)) + +with tracer.start_as_current_span("preflight") as span: + span.set_attribute("raw.input", "ping") + span.set_attribute("raw.output", "pong") + +trace_provider.force_flush() +print("sent preflight span") +``` + +**You should see** a `preflight` trace in **Observe โ†’ Traces โ†’ `support-agent`** within a few seconds. If it never lands, fix that here first โ€” see [Troubleshooting](#troubleshooting). + + + + + +The agent answers from a small inline knowledge base, so the recipe runs with no external data. The `search_help_center` tool stashes what it retrieved so you can score grounding later. + +```python +from langchain_openai import ChatOpenAI, OpenAIEmbeddings +from langchain_core.vectorstores import InMemoryVectorStore +from langchain_core.tools import tool +from langgraph.prebuilt import create_react_agent + +POLICIES = [ + "Refunds: Orders can be refunded within 30 days of delivery. Items marked 'final sale' " + "are not refundable. Refunds return to the original payment method in 5-7 business days.", + "Shipping: Standard shipping takes 3-5 business days. Express takes 1-2. We ship to the US and Canada only.", + "Account: Customers can reset a password from the login page. Support cannot see or change a password.", + "Returns: To return an item, start a return from the Orders page to get a prepaid label. " + "Returns must be shipped within 14 days of approval.", +] + +store = InMemoryVectorStore.from_texts(POLICIES, OpenAIEmbeddings()) +retriever = store.as_retriever(search_kwargs={"k": 2}) + +last_context = {"text": ""} # holds the policy retrieved this turn, for scoring + +@tool +def search_help_center(query: str) -> str: + """Search the help center for relevant policy text.""" + docs = retriever.invoke(query) + last_context["text"] = "\n\n".join(d.page_content for d in docs) + return last_context["text"] + +@tool +def lookup_order(order_id: str) -> str: + """Look up the status of an order by its ID.""" + return f"Order {order_id}: delivered on 2026-06-20, standard shipping." + +SYSTEM_PROMPT = "You are a helpful customer-support agent. Use the tools to answer." + +llm = ChatOpenAI(model="gpt-4o-mini", temperature=0) +agent = create_react_agent(llm, tools=[search_help_center, lookup_order], prompt=SYSTEM_PROMPT) +``` + + +The system prompt is deliberately minimal, so the traces contain a real failure to find. Finding it is what this cookbook is about. + + + + + + +Turn on auto-instrumentation, then run the conversations. The same `LangChainInstrumentor` captures LangGraph. Wrap each turn in `using_session` (the conversation) and `using_user` (the customer), and record the three fields an eval needs: question, answer, retrieved policy. + +```python +from fi_instrumentation import using_session, using_user +from traceai_langchain import LangChainInstrumentor + +# trace_provider and tracer come from step 1 โ€” do not register again. +LangChainInstrumentor().instrument(tracer_provider=trace_provider) + +def support_turn(session_id: str, user_id: str, question: str) -> str: + with using_session(session_id), using_user(user_id): + with tracer.start_as_current_span("support_turn") as span: + result = agent.invoke({"messages": [{"role": "user", "content": question}]}) + answer = result["messages"][-1].content + span.set_attribute("raw.input", question) + span.set_attribute("raw.output", answer) + span.set_attribute("raw.context", last_context["text"]) # retrieved policy + return answer + +# two customers, five turns between them +CONVERSATIONS = [ + ("chat_1001", "cust_42", [ + "How long do refunds take?", + "My order was a final sale, can I still get a refund?", + "Can you look up order A-3391?", + ]), + ("chat_1002", "cust_77", [ + "What shipping options do you have?", + "Can you change my password for me?", + ]), +] + +for sid, uid, questions in CONVERSATIONS: + for q in questions: + print(support_turn(sid, uid, q)) + +trace_provider.force_flush() # flush before the script exits +``` + +**You should see** two sessions in **Observe โ†’ Traces โ†’ `support-agent`**: `chat_1001` with three traces for `cust_42`, and `chat_1002` with two for `cust_77`. + + +No trace? It is almost always order or flush. `register()` and `.instrument()` must run **before** the agent call, and `force_flush()` before the process exits. + + + + + + +Before any eval runs, the traces already answer questions you'd otherwise guess at. Open the `support-agent` project: every row is one turn, with its status, model, latency, tokens, and cost. Click the final-sale turn to see the ReAct loop as a span tree โ€” the LLM call that decides to search, the `search_help_center` call with the exact policy it returned, and the LLM call that writes the answer. + +Three things are readable before any score exists: + +- **Cost** โ€” two LLM calls per answered question, and the token split between them +- **Latency** โ€” the retriever and tool spans are milliseconds; the LLM spans are the wait +- **Grounding** โ€” the tool span carries the policy text, so you can eyeball whether the answer used it + +On the final-sale turn, the retrieved policy says final-sale items are not refundable, and the answer hedges toward a refund anyway. An eval can now confirm at scale what this one trace suggests. + + + + + +Configure evals **in the platform** as an Eval Task, not from the SDK, so they run on your traces and the whole team sees the same scores. The turn span already carries `raw.input`, `raw.output`, and `raw.context` to map to. + +Create an **Eval Task** on the `support-agent` project and pick three evals, each chosen so a low score points at one component: + +- **Context Relevance** โ€” did retrieval fetch the right policy? ยท `input` โ†’ `raw.input`, `context` โ†’ `raw.context` +- **Context Adherence** โ€” did the answer stay grounded in it? ยท `output` โ†’ `raw.output`, `context` โ†’ `raw.context` +- **Completeness** โ€” did the answer fully address the question? ยท `input` โ†’ `raw.input`, `output` โ†’ `raw.output` + +Run it as **Historical** over the five turns. + + +The click-by-click โ€” creating the task, Historical vs Continuous, sampling, and mapping inputs to span attributes โ€” lives in the [Setup evals](/docs/observe/guides/setup-evals) guide. + + +**You should see** three scores per trace (illustrative shape โ€” your numbers will differ): + +| Turn | Context Relevance | Context Adherence | Completeness | +|---|---|---|---| +| "How long do refunds take?" | 0.91 | 0.88 | 0.86 | +| "Final sale, still refundable?" | 0.87 | **0.41** | 0.55 | +| "Look up order A-3391" | 0.93 | 0.90 | 0.84 | +| "What shipping options do you have?" | 0.90 | 0.85 | 0.83 | +| "Can you change my password for me?" | 0.88 | 0.82 | 0.80 | + + + + + +Filter the trace explorer to the low-scoring slice: + +```text +scores.context_adherence < 0.8 AND fi.span.kind = LLM +``` + +The final-sale turn surfaces, and its two scores split the layers: retrieval is fine (Context Relevance 0.87), grounding is not (Context Adherence 0.41). To replay the whole conversation around it, scope with `user.id = 'cust_42'` and open the session. + +Each eval's failure indicts one layer: + +| Eval signal | Insight | Where a fix lives | +|---|---|---| +| Context Relevance low | wrong policy retrieved | retrieval: chunking, query, `k`, filters | +| Relevance ok, **Adherence low** | model ignores good context | the prompt | +| Completeness low | partial answer | prompt, or retrieve more | + +**What you now know**, from one run: + +- **The prompt is the broken layer, not retrieval** โ€” the right policy was retrieved and ignored +- **It fails on policy exceptions, not routine turns** โ€” the plain refund question scored fine; the final-sale one didn't +- **It's one turn, not an outage** โ€” every other turn for `cust_42` is healthy + +That's a ticket your team can act on, not "the bot sometimes gives wrong answers". + + + + + +Scores drift as prompts, models, and traffic change. Turn this check into standing monitors so the next regression pages you, not a customer: + +- **Alert** โ€” an [Evaluation-metric alert](/docs/observe/guides/setup-alerts) on `context_adherence`, operator *Less than*, static `0.8` +- **Saved view** โ€” save the `scores.context_adherence < 0.8` filter as an "Ungrounded answers" [view](/docs/observe/guides/explore-dashboard/views), so tomorrow's check is one click +- **Regression set** โ€” the low-scoring turns are your best future test cases; harvest them so any fix has to clear them + + + + + +## Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| No trace in Observe | `register()`/`.instrument()` ran after the agent call, or the script exited before flush | Register and instrument first; `force_flush()` before exit | +| Session or user filters return nothing | The turn ran outside the `using_session` / `using_user` context | Keep the agent call inside `with using_session(...), using_user(...)` | +| Eval Task finished but no scores | It ran Historical before the traces existed, or sampling excluded them | Re-run after sending traces; widen the date range; raise sampling | +| Eval reports a missing input | The span didn't set `raw.input` / `raw.output` / `raw.context`, or the mapping points elsewhere | Set the three attributes; map each eval input to them | +| Alert never fires | Wrong metric or project type | Evaluation-metric alert on `context_adherence`; monitors work only on `observe` projects | +| Context Relevance is the low score, not Adherence | The weak spot is retrieval, not the prompt | Same method, different layer: chunking, `k`, and filters | + +## Where to go next + +You know which layer is broken and why: the prompt ignores good context on policy-exception turns. To act on that and fix the prompt systematically, continue with [Improve a prompt automatically](/docs/cookbook/quickstart/prompt-optimization). diff --git a/src/pages/docs/cookbook/quickstart/manual-tracing.mdx b/src/pages/docs/cookbook/quickstart/manual-tracing.mdx index 93132e62..9317aefb 100644 --- a/src/pages/docs/cookbook/quickstart/manual-tracing.mdx +++ b/src/pages/docs/cookbook/quickstart/manual-tracing.mdx @@ -177,7 +177,7 @@ In Tracing, filter by tags using the **Attribute** filter: select **Attribute** ![Traces filtered by tag in the Tracing dashboard](https://fi-cookbook-assets.s3.ap-south-1.amazonaws.com/quickstart/manual-tracing/step-4-tags.png) -You can combine `using_user`, `using_session`, `using_metadata`, and `using_tags` into a single `using_attributes()` call for convenience. See the [tracing reference](/docs/observe/features/manual-tracing/set-up-tracing) for details. +You can combine `using_user`, `using_session`, `using_metadata`, and `using_tags` into a single `using_attributes()` call for convenience. See the [tracing reference](/docs/sdk/tracing/set-up-tracing) for details. diff --git a/src/pages/docs/faq.mdx b/src/pages/docs/faq.mdx index 123fbbc3..108cf801 100644 --- a/src/pages/docs/faq.mdx +++ b/src/pages/docs/faq.mdx @@ -129,7 +129,7 @@ Observe captures every LLM call, tool use, and agent decision as a trace. You ca **How do I set up alerts?** -Configure alerts to notify you about anomalies based on defined thresholds. See [Alerts & Monitors](/docs/observe/features/alerts). +Configure alerts to notify you about anomalies based on defined thresholds. See [Alerts & Monitors](/docs/observe/guides/setup-alerts). --- diff --git a/src/pages/docs/get-started/send-your-first-trace.mdx b/src/pages/docs/get-started/send-your-first-trace.mdx index 5d254402..d1f49169 100644 --- a/src/pages/docs/get-started/send-your-first-trace.mdx +++ b/src/pages/docs/get-started/send-your-first-trace.mdx @@ -144,7 +144,7 @@ Not seeing your traces? Try checking these: traceAI supports Anthropic, LangChain, LlamaIndex, and 30+ more with the same four steps - + Start scoring quality on the traces you capture diff --git a/src/pages/docs/integrations/export/pagerduty.mdx b/src/pages/docs/integrations/export/pagerduty.mdx index 64f98c28..f333ddd2 100644 --- a/src/pages/docs/integrations/export/pagerduty.mdx +++ b/src/pages/docs/integrations/export/pagerduty.mdx @@ -126,7 +126,7 @@ Your routing key may have been revoked or the PagerDuty service was deleted. Gen - + diff --git a/src/pages/docs/integrations/import/langfuse.mdx b/src/pages/docs/integrations/import/langfuse.mdx index a41473ef..ed2143a6 100644 --- a/src/pages/docs/integrations/import/langfuse.mdx +++ b/src/pages/docs/integrations/import/langfuse.mdx @@ -197,6 +197,6 @@ Large backfills can hit Langfuse's API rate limits, especially on the free tier. - + diff --git a/src/pages/docs/integrations/index.mdx b/src/pages/docs/integrations/index.mdx index c61343f7..97710ecc 100644 --- a/src/pages/docs/integrations/index.mdx +++ b/src/pages/docs/integrations/index.mdx @@ -43,7 +43,7 @@ TraceAI provides pre-built auto-instrumentation for the following frameworks and - + @@ -55,8 +55,8 @@ The Langfuse card above is for SDK-level tracing integration (sending new traces - - + + ### Other diff --git a/src/pages/docs/observe/concepts/observability-model.mdx b/src/pages/docs/observe/concepts/observability-model.mdx new file mode 100644 index 00000000..a392e6ab --- /dev/null +++ b/src/pages/docs/observe/concepts/observability-model.mdx @@ -0,0 +1,112 @@ +--- +title: "Observability model" +description: "How spans, traces, sessions, and users fit together into one hierarchy." +slug: "observability-model" +page_type: "concept" +diataxis: "explanation" +products: ["Observe"] +concept_family: "observability" +concept_level: "foundational" +audience: ["engineer"] +difficulty: "beginner" +status: "review" +owner: "observability" +reviewers: ["observability-eng"] +last_tested: "2026-06-18" +last_diagram_reviewed: "2026-06-17" +schema_type: "TechArticle" +primary_question: "How do traces, spans, sessions, users, and evals relate in FutureAGI Observe?" +direct_answer: "Observe is built on a hierarchy: a span is one operation, spans sharing a trace ID form a trace (one request), traces sharing a session ID form a session, and sessions belong to a user. Eval scores attach to spans or traces. All of it is captured as OpenTelemetry data via the traceAI SDK." +seo: + title: "The FutureAGI Observe observability model" + description: "The entity hierarchy behind Observe โ€” spans, traces, sessions, users, and eval scores โ€” and how traceAI and OpenTelemetry collect it." + primary_keyword: "llm observability data model" + direct_answer: true +geo: + answer_target: "How do traces, spans, sessions, and users relate in FutureAGI Observe?" + llm_summary: "Spans nest into traces by trace ID, traces group into sessions by session ID, sessions belong to users, and eval scores attach to spans or traces โ€” all collected via traceAI over OpenTelemetry." +canonical: "/docs/observe/concepts/observability-model" +related: + - "/docs/observe/concepts/traces" + - "/docs/observe/concepts/spans" + - "/docs/observe/concepts/sessions" + - "https://opentelemetry.io/docs/" +--- + +## A few objects that nest + +Observe is built on a small set of objects that nest: a [span](/docs/observe/concepts/spans) sits inside a [trace](/docs/observe/concepts/traces), a trace inside a [session](/docs/observe/concepts/sessions), and a session belongs to a [user](/docs/observe/concepts/users). Eval scores attach on top, to a span or a whole trace. Knowing how these relate is the difference between knowing where to look and guessing: every view in Observe, from the trace list to dashboards and alerts, is a different lens on this one **hierarchy**. + +All of it is collected the same way: your app emits spans through the [traceAI](/docs/observe/concepts/traceai) SDK, which is built on [OpenTelemetry](https://opentelemetry.io/docs/), and Observe reads them. + +--- + +## Mental model + +The objects form a strict containment hierarchy, and the ID on each span is what reconstructs it. You don't assemble traces or sessions by hand; shared IDs do that automatically. + + S["Session ยท session.id"] + S --> T1["Trace ยท one request"] + S --> T2["Trace ยท one request"] + T1 --> SP1["Span ยท llm call"] + T1 --> SP2["Span ยท tool call"] + SP1 --> EV["Eval score"] + T1 --> EV2["Eval score"]`} /> + +Read it bottom-up when debugging (a bad span, up to its trace, its session, its user) and top-down when analyzing (a user's sessions, down to their traces and the spans inside). + +## Key terms + +| Object | What it is | Identified by | Learn more | +|---|---|---|---| +| **Span** | One operation โ€” an LLM call, tool call, retrieval, or agent step โ€” with input, output, timing, and cost. | Span ID (+ parent span ID) | [Spans](/docs/observe/concepts/spans) | +| **Trace** | One complete request, made of all the spans that share its trace ID. | Trace ID | [Traces](/docs/observe/concepts/traces) | +| **Session** | A multi-turn conversation โ€” the traces that share a session ID. | `session.id` | [Sessions](/docs/observe/concepts/sessions) | +| **User** | One end user, across all their sessions and traces. | `user.id` | [Users](/docs/observe/concepts/users) | +| **Eval score** | A quality score attached to a span or trace. | Attached to span/trace | [Trace evals](/docs/observe/guides/setup-evals) | +| **OpenTelemetry** | The open standard the spans are emitted in. | โ€” | [OpenTelemetry](https://opentelemetry.io/docs/) | +| **traceAI** | The SDK that produces the spans. | โ€” | [traceAI SDK](/docs/observe/concepts/traceai) | + +## How it works + +Your app emits spans through traceAI (or OpenTelemetry directly). Each span carries a trace ID, so all spans from one request form a single trace. The backend receives them over OTLP (HTTP or gRPC) and stores them by project, and from there every Observe view runs on the same data. + +To enrich the hierarchy, attach a `session.id` and a `user.id` in code, and every span inside picks them up. See [set session and user IDs](/docs/sdk/tracing/set-session-user-id) and [add attributes and metadata](/docs/sdk/tracing/add-attributes-metadata-tags). + +## Walking the hierarchy when debugging + +The hierarchy is most useful read bottom-up. A typical investigation starts at the smallest object and climbs: + +1. **Start at the span.** A customer complained the assistant gave a wrong answer. You open the span that produced it and read its real input and output: the exact prompt and completion, not a paraphrase. +2. **Climb to the trace.** The span alone rarely explains the failure. You move up to its trace and read the other spans in order: the retrieval that fed bad context, the tool call that returned stale data, the agent step that chose the wrong path. The trace is where the request becomes legible. +3. **Climb to the session.** If the request looked fine in isolation but the conversation still went wrong, you open its session and read the earlier turns. Multi-turn problems, like the assistant losing track or contradicting itself, only show up here. +4. **Climb to the user.** If the pattern repeats, you pivot to the user to see whether it is one customer's data or a systemic issue across everyone. + +Because the IDs link each level to the next, every climb is one click, and you never reassemble context by hand. The same path runs top-down for analysis: start at a user, expand their sessions, then traces, then the spans inside. + +## When to use this model + +- **Debugging a bad answer**: start at the span that produced it, then read the rest of the trace for context +- **Analyzing a conversation**: open the session to see every turn in order +- **Investigating a customer**: pivot from a user to all their sessions and traces +- **Measuring quality**: read eval scores attached at the span or trace level + +## Common mistakes + +- **Confusing a span with a trace**: a slow trace tells you a request was slow; the span tells you which step was slow. See [Spans](/docs/observe/concepts/spans) +- **Expecting sessions without a session ID**: traces only group into a session if they share a `session.id`. Set it in code +- **Looking for customers you never tagged**: per-user views need `user.id` on the spans + +## Keep exploring + + + + Send a trace and watch the model fill in + + + Score spans and traces for quality and safety + + diff --git a/src/pages/docs/observe/concepts/sessions.mdx b/src/pages/docs/observe/concepts/sessions.mdx new file mode 100644 index 00000000..c093d7b8 --- /dev/null +++ b/src/pages/docs/observe/concepts/sessions.mdx @@ -0,0 +1,48 @@ +--- +title: "Sessions" +description: "Reading a whole multi-turn conversation as one unit." +--- + +## A session is one conversation + +A **session** is one multi-turn conversation, reassembled from its [traces](/docs/observe/concepts/traces). When a chatbot answers five messages, that is five separate traces, one per turn. Give them all the same `session.id` and Observe ties them back into one conversation that sits one level above the trace: the session holds the ordered traces, and each trace holds its [spans](/docs/observe/concepts/spans). The name is [OpenTelemetry](https://opentelemetry.io/docs/)'s own, and setting it once around a turn's work carries it to every span inside, just like the trace ID. + + T1["Trace (turn 1)"] + S --> T2["Trace (turn 2)"] + S --> T3["Trace (turn 3)"] + T1 --> P1["spans"] + T2 --> P2["spans"] + T3 --> P3["spans"]`} /> + +Take a three-turn support chat. Each turn is its own request, so each is its own trace, but all three share `session.id="chat_abc"`. Observe rolls them into one row you read top to bottom, from the opening question to the resolution, with the whole conversation's duration, cost, and token count in one place. Reuse that same ID across turns and the conversation grows, a fresh ID each turn would leave you with sessions of one trace each. + +## When to use + +Reach for a session when the problem runs across turns, not a single request: the assistant that kept losing track across a conversation, someone who drops off or escalates halfway through a flow, or any time you want the whole chat's duration, cost, and tokens at once instead of per request. + +When the grain is wrong, reach elsewhere: + +- Debugging a single request: open its [trace](/docs/observe/concepts/traces), a session is too coarse +- Rolling up by person, not conversation: use [Users](/docs/observe/concepts/users), which gathers every conversation one person had +- Aggregate trends across many sessions: build a dashboard + +## Why it matters + +A conversation's problems are invisible one request at a time. Whether the assistant stayed coherent across turns, where someone gave up, what a whole support chat cost, none of it shows on a single trace. Grouping the turns into a session puts the conversation back together, so you debug and measure the thing your user actually lived through, not the fragments of it. + +## Keep exploring + + + + Roll every conversation up by the person who had it + + + Read, filter, and sort the Sessions view + + + Attach session.id in traceAI + + diff --git a/src/pages/docs/observe/concepts/spans.mdx b/src/pages/docs/observe/concepts/spans.mdx index cfbb4920..e8776757 100644 --- a/src/pages/docs/observe/concepts/spans.mdx +++ b/src/pages/docs/observe/concepts/spans.mdx @@ -5,9 +5,9 @@ description: "Pinpointing the single step behind a slow or wrong answer." ## A span is one step -A span is one operation inside a [trace](/docs/observe/concepts/traces): a single model call, tool call, retrieval, agent step, guardrail check, or evaluator run. It records its own input and output, when it started and finished, whether it succeeded, and, for model calls, the tokens and cost it ran up. Where a trace is the whole request, a span is one step inside it. +A **span** is one operation inside a [trace](/docs/observe/concepts/traces): a single model call, tool call, retrieval, agent step, guardrail check, or evaluator run. It records its own input and output, when it started and finished, whether it succeeded, and, for model calls, the tokens and cost it ran up. Where a trace is the whole request, a span is one step inside it. -Under the hood, a span is an [OpenTelemetry](/docs/tracing/concepts/otel) span. OpenTelemetry defines the shape, a named, timed unit of work with a status, a parent, and key-value attributes, and traceAI fills those attributes with LLM-specific keys: the span `kind` that says what ran, the prompt and completion, the token counts. So every span you see in Observe is a standard OTel span carrying traceAI's LLM attributes. +Under the hood, a span is an [OpenTelemetry](https://opentelemetry.io/docs/) span. OpenTelemetry defines the shape, a named, timed unit of work with a status, a parent, and key-value attributes, and traceAI fills those attributes with LLM-specific keys: the span `kind` that says what ran, the prompt and completion, the token counts. So every span you see in Observe is a standard OTel span carrying traceAI's LLM attributes. A parent span, say an agent, holds the child spans it set off, and each of those can have children of its own. That nesting is how Future AGI works out which step triggered which, and it is what lets a trace draw itself as a tree. @@ -29,15 +29,43 @@ Each box is a span with its own timing and attributes. The edges come straight f ## Why it matters -A response is only as strong as its weakest step, and that step is usually where things break. The trace tells you a request was slow or wrong; the span tells you which step did it and hands you the evidence: the exact prompt sent to the model, the arguments a tool received, the chunks a retriever pulled back, or the score an evaluator gave. traceAI captures these spans for you on supported frameworks, and where it can't reach, you can [add your own](/docs/observe/features/manual-tracing/create-tool-spans) so no part of your pipeline stays a black box. +A response is only as strong as its weakest step, and that step is usually where things break. The trace tells you a request was slow or wrong; the span tells you which step did it and hands you the evidence: the exact prompt sent to the model, the arguments a tool received, the chunks a retriever pulled back, or the score an evaluator gave. traceAI captures these spans for you on supported frameworks, and where it can't reach, you can [add your own](/docs/sdk/tracing/create-tool-spans) so no part of your pipeline stays a black box. + +## Span types + +Every span carries a `kind` that says what the operation was. Observe uses it to label the span, pick its icon, and surface the fields that matter. These are the kinds traceAI emits: + +| Type | What it represents | What it captures | +|---|---|---| +| LLM | A single model call | Model, prompt messages, completion, token counts, cost | +| Tool | A function or tool the model invoked | Tool name, arguments, and the result returned | +| Retriever | A lookup against a vector store or index | The query and the documents it returned | +| Embedding | Text turned into vectors | The input text and the embedding model | +| Reranker | Retrieved documents reordered by relevance | The documents and their new order | +| Agent | A top-level step that coordinates others | The child spans (tool calls, retrievals, model calls) it set off | +| Chain | A group of steps run as one unit | The ordered child spans it runs | +| Guardrail | A safety or policy check on an input or output | What was checked and the verdict | +| Evaluator | An eval that scores a span or trace | The metric and the score produced | + +## Span attributes + +Every span carries key-value attributes. Some come straight from [OpenTelemetry](https://opentelemetry.io/docs/) (name, status, timing, parent), and traceAI adds LLM-specific keys on top: the span kind, the prompt and completion, the model, token counts, and cost. + +- **Core (OpenTelemetry)**: name, kind, status, start and end time, parent span ID, trace ID +- **LLM (traceAI)**: model, input messages, output messages, prompt and completion tokens, cost +- **Retrieval**: query, retrieved documents and their scores +- **Grouping**: session.id, user.id, tags ## Keep exploring - + The full request that spans are grouped into - + Add custom spans where auto-instrumentation stops + + How spans, traces, sessions, and users fit together + diff --git a/src/pages/docs/observe/concepts/traceai.mdx b/src/pages/docs/observe/concepts/traceai.mdx new file mode 100644 index 00000000..f85bcfbb --- /dev/null +++ b/src/pages/docs/observe/concepts/traceai.mdx @@ -0,0 +1,45 @@ +--- +title: "traceAI" +description: "Turning your framework's model, tool, and retrieval calls into standardized spans Observe reads as traces." +--- + +## traceAI is the instrumentation SDK + +**traceAI** is Future AGI's open-source instrumentation SDK, built on [OpenTelemetry](https://opentelemetry.io/docs/). It's a set of conventions and per-framework instrumentors that capture what your AI app does (model calls, tool calls, retrievals, agent steps) and map them to standardized [span](/docs/observe/concepts/spans) attributes. Add the instrumentor for your framework, and those calls become traces in [Observe](/docs/observe) with no spans written by hand. traceAI runs inside your application and produces the spans; Observe is the product that reads them. They meet only at the span, a standard OpenTelemetry span on the wire, so the same output also works with any OTel-compatible backend. + +## Mental model + +traceAI is the adapter between your framework and OpenTelemetry. The instrumentor wraps the framework, produces standardized spans, and hands them to the OTel pipeline that exports to Future AGI. + + B["traceAI instrumentor"] + B --> C["Standardized OTel spans"] + C --> D["Future AGI Observe"]`} /> + +You pick the instrumentor that matches your framework, and the rest of the pipeline is the same OTel flow for everyone. + +## Auto and manual instrumentation + +There are two ways to produce spans, and real apps use both. Auto-instrumentation is a per-framework instrumentor that wraps a library: install the one for your framework (`traceAI-openai`, `traceAI-langchain`, and so on), call `.instrument()`, and every framework call becomes a span with no span code in your app. Manual instrumentation covers the parts no instrumentor reaches, like your own business functions, custom retrieval, or glue logic, which you wrap as [tool spans](/docs/sdk/tracing/create-tool-spans) yourself. + +Auto and manual spans feed the same provider, so they nest into one trace. That shared provider is what `register()` sets up: it builds the OpenTelemetry tracer provider, points the exporter at Future AGI, and makes it active. Nothing reaches Observe until `register()` has run, because before it there's no exporter to ship spans to. Plain non-LLM work, like a database query, needs none of this: trace it with raw OpenTelemetry and it still lands in the same trace tree. + +## Why it matters + +Raw OpenTelemetry knows nothing about LLMs. It has no concept of a prompt, a completion, token cost, or a tool call. traceAI fills that gap. It turns framework calls into LLM-shaped spans with consistent keys, so a LangChain trace and an OpenAI trace look the same in Observe and are queryable the same way. That standardization is what lets filters, evals, and dashboards work across different stacks instead of breaking every time you change frameworks. + +## Keep exploring + + + + What traceAI produces + + + Install an instrumentor and start capturing + + + Every supported framework + + diff --git a/src/pages/docs/observe/concepts/traces.mdx b/src/pages/docs/observe/concepts/traces.mdx index 251aa12e..381e09ff 100644 --- a/src/pages/docs/observe/concepts/traces.mdx +++ b/src/pages/docs/observe/concepts/traces.mdx @@ -5,7 +5,7 @@ description: "Helping you debug an AI response, step by step." ## A trace is a tree -A trace is a tree of [spans](/docs/observe/concepts/spans). The root span is the operation that kicked off the request, and every other span nests under the step that triggered it. They all share one trace ID, so the whole request stitches back together top to bottom, even when steps run across async tasks or services. +A **trace** is a tree of [spans](/docs/observe/concepts/spans). The root span is the operation that kicked off the request, and every other span nests under the step that triggered it. They all share one trace ID, so the whole request stitches back together top to bottom, even when steps run across async tasks or services. - - The individual operations a trace is built from - - + Group multiple traces into one conversation or customer - + Instrument your app so it emits traces + + How spans, traces, sessions, and users fit together + diff --git a/src/pages/docs/observe/concepts/users.mdx b/src/pages/docs/observe/concepts/users.mdx new file mode 100644 index 00000000..c7414e55 --- /dev/null +++ b/src/pages/docs/observe/concepts/users.mdx @@ -0,0 +1,51 @@ +--- +title: "Users" +description: "Following one customer across every conversation they've had." +--- + +## A user is a person across conversations + +You will often want to know *who* a conversation belonged to. A **user** is that person, the end user behind the requests. Setting a `user.id` on your spans rolls every [trace](/docs/observe/concepts/traces) and [session](/docs/observe/concepts/sessions) they generate up under one row: where a session is one conversation, a user is all of their conversations put together, so you can answer "what happened to this customer?" without writing a query. The name is [OpenTelemetry](https://opentelemetry.io/docs/)'s own, and it flows to every span in a block the same way the trace ID does. + + S1["Session"] + U --> S2["Session"] + U --> T0["Trace (no session)"] + S1 --> T1["Trace"] + S1 --> T2["Trace"] + S2 --> T3["Trace"]`} /> + +Say one customer is `user.id="cust_42"`. Every request they make carries that ID, whether it is part of a support chat or a one-off question, so Observe gathers all of it under a single row: their sessions and traces, their total cost and token use, when they first showed up and when they were last active, and how their answers scored. Open the row and you have that customer's whole history in one place. + +## When to use + +Reach for a user when the unit is a person, not a chat: a customer reports a bug and you want their entire history, a cost spike traces back to one heavy customer, you are tracking who stuck around and who dropped off (first seen, last seen, session counts), or quality is slipping for a segment and you want [eval](/docs/observe/guides/setup-evals) pass-rate per user. + +When the grain is wrong, reach elsewhere: + +- One conversation, not a person: use [Sessions](/docs/observe/concepts/sessions) +- A single request: open its [trace](/docs/observe/concepts/traces) + + +`user.id` is a grouping key, not an auth identity, and Observe never verifies it. It is exported in span data, so use a stable but non-sensitive value like a hashed customer ID, never a raw email or phone number. + + +## Why it matters + +A single trace or session shows one moment; many of the questions that matter are about the person across all their moments. Who is driving cost, who churned after a bad week, whether one segment gets worse answers than another, none of it surfaces until every request a person made rolls up together. The user is that rollup: the customer, not the request. + +## Keep exploring + + + + Group a person's traces into individual conversations + + + Read, filter, and sort the Users view + + + Attach user.id in traceAI + + diff --git a/src/pages/docs/observe/concepts/voice-observability.mdx b/src/pages/docs/observe/concepts/voice-observability.mdx new file mode 100644 index 00000000..a008b9fb --- /dev/null +++ b/src/pages/docs/observe/concepts/voice-observability.mdx @@ -0,0 +1,63 @@ +--- +title: "Voice observability" +description: "Turning every voice call into a trace you can debug and score." +--- + +## A voice call is a trace + +**Voice observability** captures each voice call as a [trace](/docs/observe/concepts/traces), the same tree of [spans](/docs/observe/concepts/spans) you get from a text app. One call becomes one trace, and each back-and-forth turn is a span inside it. The call carries the transcript, the recording, and its duration, turn count, and cost. A spoken conversation lands in the same place as every other request, ready for the same [evals](/docs/observe/guides/setup-evals), alerts, and filters. + +## Inside a voice call + +A turn is more than a single step. In an app you instrument on LiveKit or Pipecat, a turn's span breaks down into speech-to-text, the model call, and text-to-speech, so you can see exactly where a turn went wrong, not just that it did. Managed calls arrive at the turn level, and the transcript and recording sit on the call either way. + + T1["Turn ยท span"] + C --> T2["Turn ยท span"] + T1 --> STT["speech to text"] + T1 --> LLM["model call"] + T1 --> TTS["text to speech"] + C --> M["Transcript ยท recording ยท turns ยท cost"]`} /> + +Because it is an ordinary trace, a voice call fits the same [observability model](/docs/observe/concepts/observability-model) as your text traces. + +## How a call reaches Observe + +A voice call reaches Observe by one of two paths. Whichever it takes, it lands as the same trace; what differs is who produces the spans. + +| Path | For | How spans are produced | What you write | +|---|---|---|---| +| **Managed ingestion** | Hosted agents on Vapi or Retell | Observe pulls the provider's call logs | No code: connect the provider and turn observability on | +| **Auto-instrumentation** | Apps built on LiveKit or Pipecat | Your app emits a span per turn through [traceAI](/docs/observe/concepts/traceai) | A few lines of traceAI setup | + +For the managed-ingestion setup, see [Voice observability](/docs/observe/concepts/voice-observability). + +## Debugging a call + +Take a support line running on a Vapi assistant. Observe pulls each finished call in as its own trace, so you read it top to bottom and follow the conversation turn by turn. When a caller reports the agent misheard their order number, you open that one call, jump to the turn where it happened, and play the audio back, instead of guessing from a dashboard. + +## When to use + +Reach for voice observability when what you are debugging is a spoken conversation: a caller who got the wrong answer, an agent that ran long, a call that cost more than it should. It also fits when you want those calls sitting alongside the rest of your traces, ready to score and monitor. + +When the grain is wrong, reach elsewhere: + +- A text or SDK app, not a voice one: instrument it directly and start at the [quickstart](/docs/observe/quickstart) +- Trends across many calls, not one: build a dashboard + +## Why it matters + +Voice failures are the ones you hear about from a customer, not a log. A spoken call normally leaves nothing behind to inspect; capturing it as a trace changes that, so a complaint becomes a call you can open, read, and replay instead of a guess. + +## Keep exploring + + + + Score voice conversations for quality and safety + + + How spans, traces, sessions, and users fit together + + diff --git a/src/pages/docs/observe/features/quickstart.mdx b/src/pages/docs/observe/features/quickstart.mdx index f810737d..e3362531 100644 --- a/src/pages/docs/observe/features/quickstart.mdx +++ b/src/pages/docs/observe/features/quickstart.mdx @@ -79,7 +79,7 @@ This is how you connect your application to Future AGI so LLM calls are captured Use one of two options: - **Auto Instrumentor**: For supported frameworks (e.g. OpenAI). Use Future AGI's [Auto Instrumentation](/docs/tracing/auto); recommended for most apps. - - **Manual tracing**: For custom spans, use [OpenTelemetry](/docs/tracing/concepts/otel). [Learn more โ†’](/docs/observe/features/manual-tracing/set-up-tracing) + - **Manual tracing**: For custom spans, use [OpenTelemetry](/docs/tracing/concepts/otel). [Learn more โ†’](/docs/sdk/tracing/set-up-tracing) Example with the OpenAI instrumentor: install the package, instrument with your trace provider, then use the OpenAI client as usual. Traces appear in your [Observe dashboard](https://app.futureagi.com/dashboard/projects/observe). diff --git a/src/pages/docs/observe/features/session.mdx b/src/pages/docs/observe/features/session.mdx index 7644ef9a..9cde25a0 100644 --- a/src/pages/docs/observe/features/session.mdx +++ b/src/pages/docs/observe/features/session.mdx @@ -1,123 +1,86 @@ --- -title: "Group Traces by Session: Multi-turn Conversation Analysis" -description: "Group traces into sessions so you can view and analyze multi-turn conversations, chatbot flows, and per-session metrics in Observe." +title: "Explore sessions & users" +description: "Read, filter, and sort the Sessions and Users views in Observe." --- -## About +Once your spans carry `session.id` and `user.id`, Observe groups them into two views: **Sessions**, one row per conversation, and **Users**, one row per end user. This page is how to read, filter, and sort them. To attach the IDs in the first place, see [Set session and user IDs](/docs/sdk/tracing/set-session-user-id); for the concepts, see [Sessions](/docs/observe/concepts/sessions) and [Users](/docs/observe/concepts/users). -Sessions group related traces together under a single identifier. A chatbot conversation, a multi-step user journey, or any sequence of LLM calls that belong to the same flow can be tracked as one session. The Observe dashboard shows sessions with their duration, cost, and token usage so you can review the full flow, drill into individual traces, and spot where things went wrong. +## The Sessions view ---- +Open the project and switch to the **Sessions** tab. Each row is one conversation. -## When to use +Observe Sessions tab for the support-agent project listing conversations chat_1001 and chat_1002 with first and last message, duration, total cost, and trace count +*One row per conversation. Sort by total cost or total traces to find the longest or most expensive sessions.* -- **Chatbot and multi-turn flows**: Group all traces for a single conversation so you can review the full exchange and debug a specific turn. -- **User journey analysis**: Treat one user's sequence of requests as a session to understand behavior and find drop-off points. -- **Session-level metrics**: See total duration, cost, and tokens for an entire session instead of checking each trace individually. -- **Filtering and drill-down**: Filter sessions by time range, open a session to see its traces, then open a trace to see spans and eval results. +The columns roll each conversation up at a glance: ---- +| Column | What it shows | +|---|---| +| **Session Id** | The shared identifier for the conversation | +| **First Message** | The opening message | +| **Last Message** | The most recent message | +| **Duration** | How long the conversation lasted | +| **Total Cost** | Combined cost of all calls in the session | +| **Total Traces** | How many requests were part of it | -## How to - - - - For a trace to appear in a session, the span must carry a **session identifier** via the `session.id` attribute. All traces with the same session name in the same project form one session. The backend creates the session automatically when the first trace with that identifier arrives. - - - - When creating a span manually, set the attribute so the trace is attached to the session: - - - ```python Python - from fi_instrumentation import register, FITracer - - trace_provider = register( - project_type=ProjectType.OBSERVE, - project_name="PROJECT_NAME", - ) - - tracer = FITracer(trace_provider.get_tracer(__name__)) - - with tracer.start_as_current_span( - f"SPAN_NAME", - ) as span: - span.set_status(Status(StatusCode.OK)) - span.set_attribute("session.id", "session123") - span.set_attribute("input.value", "input") - span.set_attribute("output.value", "output") - ``` - ```javascript JS/TS - const { register, ProjectType } = require("@traceai/fi-core"); - - const traceProvider = register({ - projectType: ProjectType.OBSERVE, - projectName: "FUTURE_AGI" - }); - - const tracer = traceProvider.getTracer("manual-instrumentation-example"); - - tracer.startActiveSpan("HandleFunctionCall", {}, (span) => { - span.setAttribute("session.id", "my-session-id"); - span.end(); - }); - ``` - - - - - To tag all spans in a block with the same session, use context so every span gets `session.id` automatically: - - - ```python Python - from fi_instrumentation import using_session - - with using_session(session_id="my-session-id"): - # All spans created within this block get session.id = "my-session-id" - ... - ``` - ```javascript JS/TS - import { context, propagation } from "@opentelemetry/api"; - - const sessionId = "my-js-session-id"; - - const activeContext = context.active(); - const baggageWithSession = propagation.createBaggage({ - "session.id": { value: sessionId } - }); - const newContext = propagation.setBaggage(activeContext, baggageWithSession); - - context.with(newContext, () => { - // All spans created within this block get session.id = "my-js-session-id" - }); - ``` - - - - - In the Observe UI, open the project and go to the Sessions view. You can filter by time range, see a list of sessions with duration and metrics, open a session to see its traces, and click **View Trace** for span-level detail and [eval](/docs/observe/features/evals) results. - - - - - For more on setting `session.id` with Trace AI helpers and context, see the [manual tracing guide](/docs/observe/features/manual-tracing/set-session-user-id). - +Open a session for its detail view, the traces in order, each with its eval scores and annotations. From there you open any trace for the full span tree. ---- +Session detail with the Session History conversation and a Trace Details panel showing trace ID, evaluations, and metadata +*A session opened: the conversation turn by turn, with each turn's trace, evals, and metadata.* + +Narrow the list with the filter bar, by `session.id`, metadata, or any span attribute (see the [filter syntax reference](/docs/observe/reference/filters)), and scope it to a time window with the date-range picker, which recomputes the column metrics for the window. For voice and other replayable sessions, configure session replay to step back through a conversation as it happened. + +## The Users view + +Switch to the **Users** view. Each row is one end user. + +Observe Users view for the support-agent project listing end users cust_42 and cust_77 with User ID, First Active, Last Active, number of traces, and number of sessions columns +*One row per user, with trace and session counts rolled up. Sort by trace or session count to find your most active users.* + +The columns roll each user up: + +| Column | What it shows | +|---|---| +| **User ID** | The `user.id` value you set in code | +| **First Active** | When the user's first trace arrived | +| **Last Active** | When their most recent trace arrived | +| **No. of Traces** | How many traces are attributed to the user | +| **No. of Sessions** | How many conversations they had | + +Open a user for their detail view, where cost, evals, and guardrail results break down per session and trace, across a Traces tab and a Sessions tab. + +User detail Traces tab listing every trace for user cust_77 with input, output, status, and latency +*The Traces tab: every trace attributed to cust_77.* + +User detail Sessions tab listing every conversation for user cust_77 with first and last message, duration, cost, and trace count +*The Sessions tab: every conversation cust_77 had.* + +Filter and scope the same way as sessions, by `user.id`, metadata, or any span attribute, and by date range. + +## Not seeing your groupings? + +| Symptom | Cause | Fix | +|---|---|---| +| Traces not grouping | The call ran outside the `using_session` / `using_user` block, so spans never got the ID | Make the call inside the block (or a decorated function) | +| One conversation split across sessions | A different `session.id` was used on some turns | Reuse one stable string for the whole conversation | +| One person split across users | A different `user.id` was used on some requests | Reuse one stable string for that person | +| Row exists but no metrics | Spans carried the ID but no cost or token attributes | Confirm the LLM spans are auto-instrumented | + +For every way to attach the IDs, see [Set session and user IDs](/docs/sdk/tracing/set-session-user-id). -## Next Steps +## Related - - Connect the SDK and start capturing traces. + + What a session is and when to use one - - Run evaluations on your traced spans to score quality. + + What a user is and when to use one - - View activity and metrics per end user. + + Attach session.id and user.id in traceAI - - Get notified when metrics cross a threshold. + + Operators and fields for the filter bar diff --git a/src/pages/docs/observe/guides/explore-dashboard/display-options.mdx b/src/pages/docs/observe/guides/explore-dashboard/display-options.mdx new file mode 100644 index 00000000..3641cb0e --- /dev/null +++ b/src/pages/docs/observe/guides/explore-dashboard/display-options.mdx @@ -0,0 +1,72 @@ +--- +title: "Display options" +description: "Choose which columns show, how rows are grouped, and how the trace table is laid out." +--- + +Display options control how the trace explorer looks: which primary view sits above the table, which columns are visible and in what order, and how rows are grouped. This guide tunes them on the `support-agent` project from the [Observing a LangGraph agent](/docs/cookbook/observe-langgraph-agent-and-obtain-insights) cookbook. + +## Open the Display menu + +Click **Display** at the top right of the trace explorer. One menu holds the primary view, row height, columns, metrics, and grouping. + +The Display menu open over the trace table, with view tabs on top and sections for rows, columns, metrics, group, and settings +*The Display menu: view switcher on top, then rows, columns, metrics, and grouping* + +## Switch the primary view + +The tabs at the top of the menu swap the graph above the table between three views of the same traces: + +- **Graph View**: the default latency and traffic time series +- **Agent Graph**: your agent's structure as a node graph +- **Agent Path**: span flow as a Sankey diagram, sized by span count + +Display menu with the Graph View, Agent Graph, and Agent Path tabs at the top +*Switch views from the tabs at the top of the menu* + +Agent Graph draws the nodes the way your agent runs them: `support_turn` into the LangGraph nodes, the tools, and the LLM calls. + +Agent Graph view showing support_turn branching into LangGraph, agent, tools, ChatOpenAI, and VectorStoreRetriever nodes +*Agent Graph: the support agent's structure as a node graph* + +Agent Path shows the same run as flow, with each node's span count, so you can see where calls concentrate. + +Agent Path Sankey diagram showing span flow from support_turn through LangGraph, agent, tools, and the LLM and retriever spans with span counts +*Agent Path: span flow sized by how many spans ran at each step* + +## Choose your columns + +Under **Columns**, **View columns** shows or hides any column, **Autosize columns** fits them to their content, and **Add custom columns** surfaces a span attribute as its own column. You can also reorder the columns so the fields you care about come first. + +