diff --git a/public/images/docs/observe/alerts-choose-project.png b/public/images/docs/observe/alerts-choose-project.png new file mode 100644 index 00000000..53bdc1f7 Binary files /dev/null and b/public/images/docs/observe/alerts-choose-project.png differ diff --git a/public/images/docs/observe/alerts-configure.png b/public/images/docs/observe/alerts-configure.png new file mode 100644 index 00000000..9e426ac4 Binary files /dev/null and b/public/images/docs/observe/alerts-configure.png differ diff --git a/public/images/docs/observe/alerts-landing.png b/public/images/docs/observe/alerts-landing.png new file mode 100644 index 00000000..8af33fc9 Binary files /dev/null and b/public/images/docs/observe/alerts-landing.png differ diff --git a/public/images/docs/observe/alerts-select-type.png b/public/images/docs/observe/alerts-select-type.png new file mode 100644 index 00000000..7e64e128 Binary files /dev/null and b/public/images/docs/observe/alerts-select-type.png differ diff --git a/public/images/docs/observe/alerts-table.png b/public/images/docs/observe/alerts-table.png new file mode 100644 index 00000000..22e0bb6c Binary files /dev/null and b/public/images/docs/observe/alerts-table.png differ diff --git a/public/images/docs/observe/alerts-thresholds-notify.png b/public/images/docs/observe/alerts-thresholds-notify.png new file mode 100644 index 00000000..e5222f01 Binary files /dev/null and b/public/images/docs/observe/alerts-thresholds-notify.png differ diff --git a/public/images/docs/observe/dashboard-first-trace.png b/public/images/docs/observe/dashboard-first-trace.png new file mode 100644 index 00000000..076e7337 Binary files /dev/null and b/public/images/docs/observe/dashboard-first-trace.png differ diff --git a/public/images/docs/observe/dashboard-projects.png b/public/images/docs/observe/dashboard-projects.png new file mode 100644 index 00000000..b26f6d7b Binary files /dev/null and b/public/images/docs/observe/dashboard-projects.png differ diff --git a/public/images/docs/observe/dashboard-trace-detail.png b/public/images/docs/observe/dashboard-trace-detail.png new file mode 100644 index 00000000..a08b0906 Binary files /dev/null and b/public/images/docs/observe/dashboard-trace-detail.png differ diff --git a/public/images/docs/observe/dashboard-trace-table.png b/public/images/docs/observe/dashboard-trace-table.png new file mode 100644 index 00000000..af3cda47 Binary files /dev/null and b/public/images/docs/observe/dashboard-trace-table.png differ diff --git a/public/images/docs/observe/display-agent-graph.png b/public/images/docs/observe/display-agent-graph.png new file mode 100644 index 00000000..5c62a67a Binary files /dev/null and b/public/images/docs/observe/display-agent-graph.png differ diff --git a/public/images/docs/observe/display-agent-path.png b/public/images/docs/observe/display-agent-path.png new file mode 100644 index 00000000..ce682ea8 Binary files /dev/null and b/public/images/docs/observe/display-agent-path.png differ diff --git a/public/images/docs/observe/display-group-by-spans.mp4 b/public/images/docs/observe/display-group-by-spans.mp4 new file mode 100644 index 00000000..f5575647 Binary files /dev/null and b/public/images/docs/observe/display-group-by-spans.mp4 differ diff --git a/public/images/docs/observe/display-menu.png b/public/images/docs/observe/display-menu.png new file mode 100644 index 00000000..aa29a1ff Binary files /dev/null and b/public/images/docs/observe/display-menu.png differ diff --git a/public/images/docs/observe/display-reorder-columns.mp4 b/public/images/docs/observe/display-reorder-columns.mp4 new file mode 100644 index 00000000..124e8a05 Binary files /dev/null and b/public/images/docs/observe/display-reorder-columns.mp4 differ diff --git a/public/images/docs/observe/display-view-switcher.png b/public/images/docs/observe/display-view-switcher.png new file mode 100644 index 00000000..575bd915 Binary files /dev/null and b/public/images/docs/observe/display-view-switcher.png differ diff --git a/public/images/docs/observe/evals-add-button.png b/public/images/docs/observe/evals-add-button.png new file mode 100644 index 00000000..e36abfc6 Binary files /dev/null and b/public/images/docs/observe/evals-add-button.png differ diff --git a/public/images/docs/observe/evals-add-evaluation.png b/public/images/docs/observe/evals-add-evaluation.png new file mode 100644 index 00000000..555c8e1b Binary files /dev/null and b/public/images/docs/observe/evals-add-evaluation.png differ diff --git a/public/images/docs/observe/evals-configure.png b/public/images/docs/observe/evals-configure.png new file mode 100644 index 00000000..070f9b17 Binary files /dev/null and b/public/images/docs/observe/evals-configure.png differ diff --git a/public/images/docs/observe/evals-create-task.png b/public/images/docs/observe/evals-create-task.png new file mode 100644 index 00000000..c8960dcd Binary files /dev/null and b/public/images/docs/observe/evals-create-task.png differ diff --git a/public/images/docs/observe/evals-results-column.png b/public/images/docs/observe/evals-results-column.png new file mode 100644 index 00000000..d94092f4 Binary files /dev/null and b/public/images/docs/observe/evals-results-column.png differ diff --git a/public/images/docs/observe/evals-schedule.png b/public/images/docs/observe/evals-schedule.png new file mode 100644 index 00000000..d0bc4c0d Binary files /dev/null and b/public/images/docs/observe/evals-schedule.png differ diff --git a/public/images/docs/observe/evals-select-evaluation.png b/public/images/docs/observe/evals-select-evaluation.png new file mode 100644 index 00000000..4e7c46b8 Binary files /dev/null and b/public/images/docs/observe/evals-select-evaluation.png differ diff --git a/public/images/docs/observe/filters-open-panel.png b/public/images/docs/observe/filters-open-panel.png new file mode 100644 index 00000000..f80c9aaa Binary files /dev/null and b/public/images/docs/observe/filters-open-panel.png differ diff --git a/public/images/docs/observe/filters-pick-property.png b/public/images/docs/observe/filters-pick-property.png new file mode 100644 index 00000000..468dd720 Binary files /dev/null and b/public/images/docs/observe/filters-pick-property.png differ diff --git a/public/images/docs/observe/filters-pick-value.png b/public/images/docs/observe/filters-pick-value.png new file mode 100644 index 00000000..8b32730b Binary files /dev/null and b/public/images/docs/observe/filters-pick-value.png differ diff --git a/public/images/docs/observe/filters-three-modes.png b/public/images/docs/observe/filters-three-modes.png new file mode 100644 index 00000000..743a27ad Binary files /dev/null and b/public/images/docs/observe/filters-three-modes.png differ diff --git a/public/images/docs/observe/session-detail.png b/public/images/docs/observe/session-detail.png new file mode 100644 index 00000000..230039c7 Binary files /dev/null and b/public/images/docs/observe/session-detail.png differ diff --git a/public/images/docs/observe/sessions-list.png b/public/images/docs/observe/sessions-list.png new file mode 100644 index 00000000..3ca04369 Binary files /dev/null and b/public/images/docs/observe/sessions-list.png differ diff --git a/public/images/docs/observe/user-detail-sessions.png b/public/images/docs/observe/user-detail-sessions.png new file mode 100644 index 00000000..4b12119b Binary files /dev/null and b/public/images/docs/observe/user-detail-sessions.png differ diff --git a/public/images/docs/observe/user-detail-traces.png b/public/images/docs/observe/user-detail-traces.png new file mode 100644 index 00000000..abf4c7a8 Binary files /dev/null and b/public/images/docs/observe/user-detail-traces.png differ diff --git a/public/images/docs/observe/users-list.png b/public/images/docs/observe/users-list.png new file mode 100644 index 00000000..f9fed54b Binary files /dev/null and b/public/images/docs/observe/users-list.png differ diff --git a/public/images/docs/observe/views-create.png b/public/images/docs/observe/views-create.png new file mode 100644 index 00000000..4b8f6cb7 Binary files /dev/null and b/public/images/docs/observe/views-create.png differ diff --git a/public/images/docs/observe/views-save-modal.png b/public/images/docs/observe/views-save-modal.png new file mode 100644 index 00000000..980b9356 Binary files /dev/null and b/public/images/docs/observe/views-save-modal.png differ diff --git a/public/images/docs/observe/views-saved-tab.png b/public/images/docs/observe/views-saved-tab.png new file mode 100644 index 00000000..9bafcd0d Binary files /dev/null and b/public/images/docs/observe/views-saved-tab.png differ diff --git a/src/components/Sidebar.astro b/src/components/Sidebar.astro index 2df4e379..ad2ec008 100644 --- a/src/components/Sidebar.astro +++ b/src/components/Sidebar.astro @@ -10,6 +10,7 @@ import { tabNavigation, getActiveTab, getActiveGroup, type NavItem, type NavGrou const currentPath = Astro.url.pathname; const activeTab = getActiveTab(currentPath); const isDocsTab = activeTab?.tab === 'Docs'; +const isSdkTab = activeTab?.tab === 'SDK'; // For Docs tab: groups are sections you switch between // For other tabs: groups are also switchable sections @@ -21,14 +22,23 @@ let activeGroup: NavGroup | undefined; if (isDocsTab) { activeGroup = getActiveGroup(currentPath); } else { - // For non-Docs tabs, find the group containing current page - activeGroup = allGroups.find(g => - g.items.some(item => { - const norm = currentPath.replace(/\/$/, '') || '/'; - const href = (item.href || '').replace(/\/$/, '') || '/'; - return href === norm || (href !== '/' && norm.startsWith(href + '/')); - }) - ) || allGroups[0]; + // For non-Docs tabs, pick the group whose deepest matching item href is the + // longest (most specific), so a broad landing group (e.g. SDK Overview at + // /docs/sdk) never shadows a product group like Evaluation (/docs/sdk/evals). + const norm = currentPath.replace(/\/$/, '') || '/'; + let bestLen = -1; + const consider = (g: NavGroup, list: NavItem[]) => { + for (const it of list) { + if (it.href) { + const href = it.href.replace(/\/$/, '') || '/'; + const matches = href === norm || (href !== '/' && href !== '/docs' && norm.startsWith(href + '/')); + if (matches && href.length > bestLen) { bestLen = href.length; activeGroup = g; } + } + if (it.items) consider(g, it.items); + } + }; + allGroups.forEach(g => consider(g, g.items)); + if (!activeGroup) activeGroup = allGroups[0]; } const items = activeGroup?.items || []; @@ -127,6 +137,22 @@ const referenceItems: DropdownItem[] = [ { title: 'API Reference', icon: 'webhook', href: '/docs/api' }, ]; +// SDK tab: the same phase-grouped product dropdown as Docs, with the SDK's products. +const sdkPhasedOrder: { phase: string; titles: string[] }[] = [ + { phase: 'Start', titles: ['SDK Overview'] }, + { phase: 'Observe & diagnose', titles: ['traceAI'] }, + { phase: 'Evaluate & measure', titles: ['Evaluation', 'Simulation', 'Datasets'] }, + { phase: 'Improve', titles: ['Prompt Optimization', 'Annotation Queues'] }, + { phase: 'Build & connect', titles: ['Knowledge Base'] }, + { phase: 'Protect', titles: ['Protect'] }, +]; +const sdkReferenceItems: DropdownItem[] = [ + { title: 'Product Docs', icon: 'book', href: '/docs' }, + { title: 'Integrations', icon: 'plug', href: '/docs/integrations' }, + { title: 'Guides', icon: 'compass', href: '/docs/cookbook' }, + { title: 'API Reference', icon: 'webhook', href: '/docs/api' }, +]; + let dropdownSections: DropdownSection[]; if (isDocsTab) { dropdownSections = phasedOrder @@ -140,6 +166,17 @@ if (isDocsTab) { if (leftovers.length) dropdownSections.push({ phase: 'More', items: leftovers }); dropdownSections.push({ phase: 'Reference', items: referenceItems }); +} else if (isSdkTab) { + dropdownSections = sdkPhasedOrder + .map(p => ({ phase: p.phase, items: p.titles.filter(t => groupLookup.has(t)).map(toItem) })) + .filter(s => s.items.length > 0); + + // Future-proofing: any SDK group not placed above still shows under "More". + const placed = new Set(sdkPhasedOrder.flatMap(p => p.titles)); + const leftovers = allGroups.map(g => g.group).filter(t => !placed.has(t)).map(toItem); + if (leftovers.length) dropdownSections.push({ phase: 'More', items: leftovers }); + + dropdownSections.push({ phase: 'Reference', items: sdkReferenceItems }); } else { // Other multi-group tabs: one unlabelled section (empty phase hides the header). dropdownSections = [{ phase: '', items: allGroups.map(g => toItem(g.group)) }]; diff --git a/src/lib/navigation.ts b/src/lib/navigation.ts index e9b8315a..df36dfb7 100644 --- a/src/lib/navigation.ts +++ b/src/lib/navigation.ts @@ -60,13 +60,28 @@ export const tabNavigation: NavTab[] = [ title: 'Self-Hosting', items: [ { title: 'Overview', href: '/docs/self-hosting' }, - { title: 'System requirements', href: '/docs/self-hosting/requirements' }, - { title: 'Environment variables', href: '/docs/self-hosting/environment' }, - { title: 'Configuration', href: '/docs/self-hosting/configuration' }, - { title: 'Docker Compose', href: '/docs/self-hosting/docker-compose' }, - { title: 'Production', href: '/docs/self-hosting/production' }, - { title: 'User management', href: '/docs/self-hosting/user-management' }, - { title: 'Troubleshooting and FAQs', href: '/docs/self-hosting/troubleshooting' }, + { title: 'Requirements', href: '/docs/self-hosting/requirements' }, + { title: 'Installation', href: '/docs/self-hosting/installation' }, + { + title: 'Configuration', + items: [ + { title: 'System configuration', href: '/docs/self-hosting/configuration/system' }, + { title: 'Environment variables', href: '/docs/self-hosting/configuration/environment' }, + ] + }, + { + title: 'Production', + items: [ + { title: 'Overview', href: '/docs/self-hosting/production' }, + { title: 'Checklist', href: '/docs/self-hosting/production/checklist' }, + { title: 'Security & TLS', href: '/docs/self-hosting/production/security-tls' }, + { title: 'Backups & restore', href: '/docs/self-hosting/production/backups-restore' }, + { title: 'Monitoring', href: '/docs/self-hosting/production/monitoring' }, + { title: 'Upgrades & rollback', href: '/docs/self-hosting/production/upgrades-rollback' }, + ] + }, + { title: 'Troubleshooting & FAQs', href: '/docs/self-hosting/troubleshooting' }, + { title: 'Support', href: '/docs/self-hosting/support' }, ] }, { @@ -380,7 +395,6 @@ export const tabNavigation: NavTab[] = [ }, { title: 'Setup alerts', href: '/docs/observe/guides/setup-alerts' }, { title: 'Setup evals', href: '/docs/observe/guides/setup-evals' }, - { title: 'Explore sessions & users', href: '/docs/observe/features/session' }, ] }, { @@ -824,7 +838,7 @@ export const tabNavigation: NavTab[] = [ title: 'Observability', icon: 'eye', items: [ - { title: 'Improving a LangGraph agent via observability', href: '/docs/cookbook/improve-langgraph-agent-with-observability' }, + { title: 'Observing a LangGraph agent and obtaining insights', href: '/docs/cookbook/observe-langgraph-agent-and-obtain-insights' }, { title: 'Text-to-SQL Evaluation', href: '/docs/cookbook/text-to-sql' }, ] }, @@ -875,51 +889,143 @@ export const tabNavigation: NavTab[] = [ href: '/docs/sdk', groups: [ { - group: 'SDK Reference', + group: 'SDK Overview', + icon: 'code', items: [ { title: 'SDK Overview', href: '/docs/sdk' }, { - title: 'AI Evaluation', + title: 'List of SDKs', items: [ - { title: 'Overview', href: '/docs/sdk/evals' }, - { title: 'Running Evaluations', href: '/docs/sdk/evals/evaluate' }, - { title: 'Distributed Evaluator', href: '/docs/sdk/evals/distributed' }, - { title: 'AutoEval', href: '/docs/sdk/evals/autoeval' }, - { title: 'Guardrails', href: '/docs/sdk/evals/guardrails-module' }, - { title: 'Local & Hybrid', href: '/docs/sdk/evals/local' }, - { title: 'OpenTelemetry', href: '/docs/sdk/evals/otel' }, - { title: 'Code Security', href: '/docs/sdk/evals/code-security' }, - { - title: 'Metrics Reference', - items: [ - { title: 'Overview', href: '/docs/sdk/evals/metrics' }, - { title: 'String & Similarity', href: '/docs/sdk/evals/metrics/string' }, - { title: 'JSON & Structured', href: '/docs/sdk/evals/metrics/json' }, - { title: 'Hallucination', href: '/docs/sdk/evals/metrics/hallucination' }, - { title: 'RAG', href: '/docs/sdk/evals/metrics/rag' }, - { title: 'Agents & Functions', href: '/docs/sdk/evals/metrics/agents' }, - { title: 'Guardrails', href: '/docs/sdk/evals/metrics/guardrails' }, - ] - }, - { title: 'Cloud Evals', href: '/docs/sdk/evals/cloud-evals' }, - { title: 'LLM-as-Judge', href: '/docs/sdk/evals/llm-judge' }, - { title: 'Streaming', href: '/docs/sdk/evals/streaming' }, - { title: 'Feedback Loops', href: '/docs/sdk/evals/feedback' }, + { title: 'Evaluation', href: '/docs/sdk/list/evaluation' }, + { title: 'TraceAI', href: '/docs/sdk/list/traceai' }, + { title: 'Core SDKs', href: '/docs/sdk/list/core' }, + ] + }, + ] + }, + { + group: 'Evaluation', + icon: 'chart', + items: [ + { title: 'Overview', href: '/docs/sdk/evals' }, + { title: 'Running Evaluations', href: '/docs/sdk/evals/evaluate' }, + { title: 'AutoEval', href: '/docs/sdk/evals/autoeval' }, + { title: 'LLM-as-Judge', href: '/docs/sdk/evals/llm-judge' }, + { title: 'Guardrails', href: '/docs/sdk/evals/guardrails-module' }, + { title: 'Local & Hybrid', href: '/docs/sdk/evals/local' }, + { title: 'Distributed Evaluator', href: '/docs/sdk/evals/distributed' }, + { title: 'Streaming', href: '/docs/sdk/evals/streaming' }, + { title: 'Cloud Evals', href: '/docs/sdk/evals/cloud-evals' }, + { title: 'Feedback Loops', href: '/docs/sdk/evals/feedback' }, + { title: 'Code Security', href: '/docs/sdk/evals/code-security' }, + { title: 'OpenTelemetry', href: '/docs/sdk/evals/otel' }, + { + title: 'Metrics Reference', + items: [ + { title: 'Overview', href: '/docs/sdk/evals/metrics' }, + { title: 'String & Similarity', href: '/docs/sdk/evals/metrics/string' }, + { title: 'JSON & Structured', href: '/docs/sdk/evals/metrics/json' }, + { title: 'Hallucination', href: '/docs/sdk/evals/metrics/hallucination' }, + { title: 'RAG', href: '/docs/sdk/evals/metrics/rag' }, + { title: 'Agents & Functions', href: '/docs/sdk/evals/metrics/agents' }, + { title: 'Guardrails', href: '/docs/sdk/evals/metrics/guardrails' }, + ] + }, + ] + }, + { + group: 'traceAI', + icon: 'eye', + items: [ + { title: 'Overview', href: '/docs/sdk/tracing' }, + { + title: 'How-to guides', + items: [ + { title: 'Set up tracing', href: '/docs/sdk/tracing/set-up-tracing' }, + { title: 'Instrument with helpers', href: '/docs/sdk/tracing/instrument-with-traceai-helpers' }, + { title: 'Set session & user IDs', href: '/docs/sdk/tracing/set-session-user-id' }, + { title: 'Attributes, metadata & tags', href: '/docs/sdk/tracing/add-attributes-metadata-tags' }, + { title: 'Log prompt templates', href: '/docs/sdk/tracing/log-prompt-templates' }, + { title: 'Events, exceptions & status', href: '/docs/sdk/tracing/add-events-exceptions-status' }, + { title: 'Mask attributes', href: '/docs/sdk/tracing/mask-span-attributes' }, + { title: 'Create tool spans', href: '/docs/sdk/tracing/create-tool-spans' }, + { title: 'Get span context', href: '/docs/sdk/tracing/get-current-span-context' }, + { title: 'In-line evals', href: '/docs/sdk/tracing/in-line-evals' }, + { title: 'Annotate via API', href: '/docs/sdk/tracing/annotating-using-api' }, + { title: 'Advanced examples', href: '/docs/sdk/tracing/advanced-tracing-examples' }, + { title: 'Langfuse integration', href: '/docs/sdk/tracing/langfuse-integration' }, + ] + }, + { + title: 'Reference', + items: [ + { title: 'register()', href: '/docs/sdk/tracing/register' }, + { title: 'FITracer & custom spans', href: '/docs/sdk/tracing/fitracer' }, + { title: 'Context helpers', href: '/docs/sdk/tracing/context-helpers' }, + { title: 'TraceConfig', href: '/docs/sdk/tracing/trace-config' }, + { title: 'EvalTags', href: '/docs/sdk/tracing/eval-tags' }, + { title: 'Instrumentors', href: '/docs/sdk/tracing/instrumentors' }, + { title: 'Environment variables', href: '/docs/sdk/tracing/environment-variables' }, + { title: 'Semantic conventions', href: '/docs/sdk/tracing/semantic-conventions' }, ] }, + ] + }, + { + group: 'Simulation', + icon: 'play', + items: [ + { title: 'Overview', href: '/docs/sdk/simulate' }, + ] + }, + { + group: 'Datasets', + icon: 'table', + items: [ + { title: 'Overview', href: '/docs/sdk/datasets' }, + ] + }, + { + group: 'Prompt Optimization', + icon: 'gauge', + items: [ + { title: 'Overview', href: '/docs/sdk/optimization' }, + ] + }, + { + group: 'Annotation Queues', + icon: 'pen', + items: [ + { title: 'Overview', href: '/docs/sdk/annotation-queues' }, { - title: 'Core SDK', + title: 'Concepts', items: [ - { title: 'Datasets', href: '/docs/sdk/datasets' }, - { title: 'Tracing', href: '/docs/sdk/tracing' }, - { title: 'Protect', href: '/docs/sdk/protect' }, - { title: 'Knowledge Base', href: '/docs/sdk/knowledgebase' }, - { title: 'Annotation Queues', href: '/docs/sdk/annotation-queues' }, - { title: 'Prompt Optimization', href: '/docs/sdk/optimization' }, - { title: 'Simulation Testing', href: '/docs/sdk/simulate' }, + { title: 'Labels', href: '/docs/sdk/annotation-queues/labels' }, + { title: 'Queue management', href: '/docs/sdk/annotation-queues/queues' }, + { title: 'Queue lifecycle', href: '/docs/sdk/annotation-queues/lifecycle' }, + { title: 'Queue items', href: '/docs/sdk/annotation-queues/items' }, + { title: 'Annotations', href: '/docs/sdk/annotation-queues/annotations' }, + { title: 'Scores', href: '/docs/sdk/annotation-queues/scores' }, + { title: 'Progress & analytics', href: '/docs/sdk/annotation-queues/analytics' }, + { title: 'Export', href: '/docs/sdk/annotation-queues/export' }, + { title: 'Data models', href: '/docs/sdk/annotation-queues/data-models' }, ] }, ] + }, + { + group: 'Knowledge Base', + icon: 'brain', + items: [ + { title: 'Overview', href: '/docs/sdk/knowledgebase' }, + ] + }, + { + group: 'Protect', + icon: 'shield', + items: [ + { title: 'Overview', href: '/docs/sdk/protect' }, + ] } ] }, diff --git a/src/lib/redirects.ts b/src/lib/redirects.ts index 430aee47..3e265470 100644 --- a/src/lib/redirects.ts +++ b/src/lib/redirects.ts @@ -1,7 +1,23 @@ // Auto-generated redirect map: old Mintlify URLs → new docs URLs // 275 redirects from futureagi.mintlify.app export const redirectMap: Record = { - '/docs/cookbook/observability': '/docs/cookbook/improve-langgraph-agent-with-observability', + // Manual-instrumentation pages moved from Observe features into the traceAI SDK section + '/docs/observe/features/manual-tracing/set-up-tracing': '/docs/sdk/tracing/set-up-tracing', + '/docs/observe/features/manual-tracing/instrument-with-traceai-helpers': '/docs/sdk/tracing/instrument-with-traceai-helpers', + '/docs/observe/features/manual-tracing/set-session-user-id': '/docs/sdk/tracing/set-session-user-id', + '/docs/observe/features/manual-tracing/add-attributes-metadata-tags': '/docs/sdk/tracing/add-attributes-metadata-tags', + '/docs/observe/features/manual-tracing/log-prompt-templates': '/docs/sdk/tracing/log-prompt-templates', + '/docs/observe/features/manual-tracing/add-events-exceptions-status': '/docs/sdk/tracing/add-events-exceptions-status', + '/docs/observe/features/manual-tracing/mask-span-attributes': '/docs/sdk/tracing/mask-span-attributes', + '/docs/observe/features/manual-tracing/create-tool-spans': '/docs/sdk/tracing/create-tool-spans', + '/docs/observe/features/manual-tracing/get-current-span-context': '/docs/sdk/tracing/get-current-span-context', + '/docs/observe/features/manual-tracing/in-line-evals': '/docs/sdk/tracing/in-line-evals', + '/docs/observe/features/manual-tracing/annotating-using-api': '/docs/sdk/tracing/annotating-using-api', + '/docs/observe/features/manual-tracing/semantic-conventions': '/docs/sdk/tracing/semantic-conventions', + '/docs/observe/features/manual-tracing/advanced-tracing-examples': '/docs/sdk/tracing/advanced-tracing-examples', + '/docs/observe/features/manual-tracing/langfuse-integration': '/docs/sdk/tracing/langfuse-integration', + '/docs/cookbook/observability': '/docs/cookbook/observe-langgraph-agent-and-obtain-insights', + '/docs/cookbook/improve-langgraph-agent-with-observability': '/docs/cookbook/observe-langgraph-agent-and-obtain-insights', '/docs/observe/features/annotation-queue-using-sdk': '/docs/annotations/sdk/annotation-queue-using-sdk', '/docs/observe/voice/set-up': '/docs/observe/features/voice', '/docs/quickstart/installation': '/docs/installation', @@ -15,17 +31,17 @@ export const redirectMap: Record = { '/docs/knowledge-base/concept': '/docs/knowledge-base/concepts/concept', '/docs/prompt-workbench': '/docs/prompt', '/docs/prompt-workbench/sdk': '/docs/prompt/features/sdk', - '/docs/tracing/manual/log-prompt-templates': '/docs/observe/features/manual-tracing/log-prompt-templates', - '/docs/tracing/manual/in-line-evals': '/docs/observe/features/manual-tracing/in-line-evals', + '/docs/tracing/manual/log-prompt-templates': '/docs/sdk/tracing/log-prompt-templates', + '/docs/tracing/manual/in-line-evals': '/docs/sdk/tracing/in-line-evals', '/docs/simulation/set-up/scenarios': '/docs/simulation/concepts/scenarios', '/docs/simulation/set-up/agent-definition': '/docs/simulation/concepts/agent-definition', '/docs/tracing/concepts/components': '/docs/tracing/concepts', - '/docs/tracing/manual/add-attributes-metadata-tags': '/docs/observe/features/manual-tracing/add-attributes-metadata-tags', - '/docs/tracing/manual/add-events-exceptions-status': '/docs/observe/features/manual-tracing/add-events-exceptions-status', - '/docs/tracing/manual/advanced-tracing-examples': '/docs/observe/features/manual-tracing/advanced-tracing-examples', - '/docs/tracing/manual/langfuse-integration': '/docs/observe/features/manual-tracing/langfuse-integration', - '/docs/tracing/manual/set-session-user-id': '/docs/observe/features/manual-tracing/set-session-user-id', - '/docs/tracing/manual/set-up-tracing': '/docs/observe/features/manual-tracing/set-up-tracing', + '/docs/tracing/manual/add-attributes-metadata-tags': '/docs/sdk/tracing/add-attributes-metadata-tags', + '/docs/tracing/manual/add-events-exceptions-status': '/docs/sdk/tracing/add-events-exceptions-status', + '/docs/tracing/manual/advanced-tracing-examples': '/docs/sdk/tracing/advanced-tracing-examples', + '/docs/tracing/manual/langfuse-integration': '/docs/sdk/tracing/langfuse-integration', + '/docs/tracing/manual/set-session-user-id': '/docs/sdk/tracing/set-session-user-id', + '/docs/tracing/manual/set-up-tracing': '/docs/sdk/tracing/set-up-tracing', '/docs/prism': '/docs/command-center', '/docs/prism/concepts/api-reference': '/docs/command-center/concepts/api-reference', '/docs/prism/concepts/configuration': '/docs/command-center/concepts/configuration', @@ -78,7 +94,7 @@ export const redirectMap: Record = { '/cookbook/cookbook5/How-to-build-and-incrementally-improve-RAG-applications-in-Langchain': '/docs/cookbook/rag-langchain', '/cookbook/cookbook6/How-to-evaluate-RAG-Applications': '/docs/cookbook/evaluate-rag', '/cookbook/cookbook7/Creating-Trustworthy-RAGs-for-Chatbots': '/docs/cookbook/trustworthy-rag', - '/cookbook/cookbook8/How-To-Implement-Observability': '/docs/cookbook/improve-langgraph-agent-with-observability', + '/cookbook/cookbook8/How-To-Implement-Observability': '/docs/cookbook/observe-langgraph-agent-and-obtain-insights', '/cookbook/cookbook9/How-To-Decrease-RAG-Hallucination': '/docs/cookbook/decrease-hallucination', '/cookbook/integrations/mongodb': '/docs/cookbook/mongodb', '/cookbook/optimization/basic-prompt-optimization': '/docs/cookbook/basic-optimization', @@ -164,20 +180,20 @@ export const redirectMap: Record = { '/future-agi/get-started/knowledge-base/how-to/create-kb-using-sdk': '/docs/knowledge-base/features/sdk', '/future-agi/get-started/knowledge-base/how-to/create-kb-using-ui': '/docs/knowledge-base/features/ui', '/future-agi/get-started/knowledge-base/overview': '/docs/knowledge-base', - '/future-agi/get-started/observability/manual-tracing/add-attributes-metadata-tags': '/docs/observe/features/manual-tracing/add-attributes-metadata-tags', - '/future-agi/get-started/observability/manual-tracing/add-events-exceptions-status': '/docs/observe/features/manual-tracing/add-events-exceptions-status', - '/future-agi/get-started/observability/manual-tracing/advanced-tracing-examples': '/docs/observe/features/manual-tracing/advanced-tracing-examples', - '/future-agi/get-started/observability/manual-tracing/annotating-using-api': '/docs/observe/features/manual-tracing/annotating-using-api', - '/future-agi/get-started/observability/manual-tracing/create-tool-spans': '/docs/observe/features/manual-tracing/create-tool-spans', - '/future-agi/get-started/observability/manual-tracing/get-current-span-context': '/docs/observe/features/manual-tracing/get-current-span-context', - '/future-agi/get-started/observability/manual-tracing/in-line-evals': '/docs/observe/features/manual-tracing/in-line-evals', - '/future-agi/get-started/observability/manual-tracing/instrument-with-traceai-helpers': '/docs/observe/features/manual-tracing/instrument-with-traceai-helpers', - '/future-agi/get-started/observability/manual-tracing/langfuse-intergation': '/docs/observe/features/manual-tracing/langfuse-integration', - '/future-agi/get-started/observability/manual-tracing/log-prompt-templates': '/docs/observe/features/manual-tracing/log-prompt-templates', - '/future-agi/get-started/observability/manual-tracing/mask-span-attributes': '/docs/observe/features/manual-tracing/mask-span-attributes', - '/future-agi/get-started/observability/manual-tracing/semantic-conventions': '/docs/observe/features/manual-tracing/semantic-conventions', - '/future-agi/get-started/observability/manual-tracing/set-session-user-id': '/docs/observe/features/manual-tracing/set-session-user-id', - '/future-agi/get-started/observability/manual-tracing/set-up-tracing': '/docs/observe/features/manual-tracing/set-up-tracing', + '/future-agi/get-started/observability/manual-tracing/add-attributes-metadata-tags': '/docs/sdk/tracing/add-attributes-metadata-tags', + '/future-agi/get-started/observability/manual-tracing/add-events-exceptions-status': '/docs/sdk/tracing/add-events-exceptions-status', + '/future-agi/get-started/observability/manual-tracing/advanced-tracing-examples': '/docs/sdk/tracing/advanced-tracing-examples', + '/future-agi/get-started/observability/manual-tracing/annotating-using-api': '/docs/sdk/tracing/annotating-using-api', + '/future-agi/get-started/observability/manual-tracing/create-tool-spans': '/docs/sdk/tracing/create-tool-spans', + '/future-agi/get-started/observability/manual-tracing/get-current-span-context': '/docs/sdk/tracing/get-current-span-context', + '/future-agi/get-started/observability/manual-tracing/in-line-evals': '/docs/sdk/tracing/in-line-evals', + '/future-agi/get-started/observability/manual-tracing/instrument-with-traceai-helpers': '/docs/sdk/tracing/instrument-with-traceai-helpers', + '/future-agi/get-started/observability/manual-tracing/langfuse-intergation': '/docs/sdk/tracing/langfuse-integration', + '/future-agi/get-started/observability/manual-tracing/log-prompt-templates': '/docs/sdk/tracing/log-prompt-templates', + '/future-agi/get-started/observability/manual-tracing/mask-span-attributes': '/docs/sdk/tracing/mask-span-attributes', + '/future-agi/get-started/observability/manual-tracing/semantic-conventions': '/docs/sdk/tracing/semantic-conventions', + '/future-agi/get-started/observability/manual-tracing/set-session-user-id': '/docs/sdk/tracing/set-session-user-id', + '/future-agi/get-started/observability/manual-tracing/set-up-tracing': '/docs/sdk/tracing/set-up-tracing', '/future-agi/get-started/optimization/dataset-optimization': '/docs/cookbook/quickstart/dataset-optimization', '/future-agi/get-started/optimization/how-to/using-python-sdk': '/docs/optimization/features/using-python-sdk', '/future-agi/get-started/optimization/optimizers/bayesian-search': '/docs/optimization/optimizers/bayesian-search', @@ -307,4 +323,5 @@ export const redirectMap: Record = { '/sdk-reference/python-sdk-client': '/docs/sdk', '/sdk-reference/testcase': '/docs/sdk/testcase', '/sdk-reference/tracing': '/docs/sdk/tracing', + '/docs/self-hosting/environment': '/docs/self-hosting/configuration/environment', }; diff --git a/src/pages/docs/cookbook/improve-langgraph-agent-with-observability.mdx b/src/pages/docs/cookbook/improve-langgraph-agent-with-observability.mdx deleted file mode 100644 index 5b18e132..00000000 --- a/src/pages/docs/cookbook/improve-langgraph-agent-with-observability.mdx +++ /dev/null @@ -1,251 +0,0 @@ ---- -title: "Improving a LangGraph agent via observability" -description: "Instrument a LangGraph support agent with Future AGI, score it with an Observe Eval Task, find the turn that fails, fix it from the eval signal, and prove the gain." ---- - - -Build a multi-turn LangGraph support agent, instrument it with [traceAI](/docs/observe/concepts/traceai) so every conversation lands in [Observe](/docs/observe) grouped by [session](/docs/observe/concepts/sessions) and [user](/docs/observe/concepts/users), score it with an Observe [Eval Task](/docs/observe/features/evals), then read the scores to find *where* it fails, fix that one thing in your code, and watch the number move. - - -| Time | Difficulty | Cost | -|------|------------|------| -| 30-40 min | Intermediate | ~$1 in model calls | - -The loop you'll run: - - B["Instrument with traceAI"] - B --> C["Score with an Eval Task"] - C --> D["Diagnose the failing turn"] - D --> E["Fix the prompt, re-run"] - E --> F["Confirm the gain, then alert"]`} /> - - -- FutureAGI account → [app.futureagi.com](https://app.futureagi.com) -- API keys: `FI_API_KEY` and `FI_SECRET_KEY` (see [Get your API keys](/docs/admin-settings)) -- An `OPENAI_API_KEY` -- Python 3.11 - - -## Install - -```bash -pip install fi-instrumentation-otel traceAI-langchain langgraph langchain-openai langchain-core -``` - -```bash -export FI_API_KEY="your-api-key" -export FI_SECRET_KEY="your-secret-key" -export OPENAI_API_KEY="your-openai-api-key" -``` - -## Tutorial - - - - - -Real support answers are only as good as the policy behind them. Keep the knowledge base inline so the recipe runs with no external data, then wire a prebuilt LangGraph ReAct agent over it with two tools. The `search_help_center` tool stashes what it retrieved so you can score grounding later. - -```python -from langchain_openai import ChatOpenAI, OpenAIEmbeddings -from langchain_core.vectorstores import InMemoryVectorStore -from langchain_core.tools import tool -from langgraph.prebuilt import create_react_agent - -POLICIES = [ - "Refunds: Orders can be refunded within 30 days of delivery. Items marked 'final sale' " - "are not refundable. Refunds return to the original payment method in 5-7 business days.", - "Shipping: Standard shipping takes 3-5 business days. Express takes 1-2. We ship to the US and Canada only.", - "Account: Customers can reset a password from the login page. Support cannot see or change a password.", - "Returns: To return an item, start a return from the Orders page to get a prepaid label. " - "Returns must be shipped within 14 days of approval.", -] - -store = InMemoryVectorStore.from_texts(POLICIES, OpenAIEmbeddings()) -retriever = store.as_retriever(search_kwargs={"k": 2}) - -last_context = {"text": ""} # holds the policy retrieved this turn, for scoring - -@tool -def search_help_center(query: str) -> str: - """Search the help center for relevant policy text.""" - docs = retriever.invoke(query) - last_context["text"] = "\n\n".join(d.page_content for d in docs) - return last_context["text"] - -@tool -def lookup_order(order_id: str) -> str: - """Look up the status of an order by its ID.""" - return f"Order {order_id}: delivered on 2026-06-20, standard shipping." - -SYSTEM_PROMPT = "You are a helpful customer-support agent. Use the tools to answer." - -llm = ChatOpenAI(model="gpt-4o-mini", temperature=0) -agent = create_react_agent(llm, tools=[search_help_center, lookup_order], prompt=SYSTEM_PROMPT) -``` - - -The `SYSTEM_PROMPT` is deliberately weak. It is the thing you'll fix later, once the evals tell you to. - - - - - - -Register tracing against an **Observe** project and turn on auto-instrumentation. The same `LangChainInstrumentor` captures LangGraph, there is no separate instrumentor. Wrap each turn in `using_session` (the conversation) and `using_user` (the customer), and in one span record the three fields an eval will need: the question, the answer, and the retrieved policy. - -```python -from fi_instrumentation import register, using_session, using_user, FITracer -from fi_instrumentation.fi_types import ProjectType -from traceai_langchain import LangChainInstrumentor - -trace_provider = register( - project_type=ProjectType.OBSERVE, - project_name="support-agent", - set_global_tracer_provider=True, -) -LangChainInstrumentor().instrument(tracer_provider=trace_provider) -tracer = FITracer(trace_provider.get_tracer(__name__)) - -def support_turn(session_id: str, user_id: str, question: str) -> str: - with using_session(session_id), using_user(user_id): - with tracer.start_as_current_span("support_turn") as span: - result = agent.invoke({"messages": [{"role": "user", "content": question}]}) - answer = result["messages"][-1].content - span.set_attribute("raw.input", question) - span.set_attribute("raw.output", answer) - span.set_attribute("raw.context", last_context["text"]) # retrieved policy - return answer - -# two customers, five turns between them -CONVERSATIONS = [ - ("chat_1001", "cust_42", [ - "How long do refunds take?", - "My order was a final sale, can I still get a refund?", - "Can you look up order A-3391?", - ]), - ("chat_1002", "cust_77", [ - "What shipping options do you have?", - "Can you change my password for me?", - ]), -] - -for sid, uid, questions in CONVERSATIONS: - for q in questions: - print(support_turn(sid, uid, q)) - -trace_provider.force_flush() # BatchSpanProcessor buffers up to 5s; flush before a short script exits -``` - -**You should see** five answers in the terminal, and within a few seconds, in **Observe > Traces > `support-agent`**, two sessions: `chat_1001` holding three traces for user `cust_42`, and `chat_1002` holding two for `cust_77`. The final-sale refund turn is the one to watch. - - -No trace? It is almost always order or flush. `register()` and `.instrument()` must run **before** the agent call, and `force_flush()` must run before the process exits. - - - - - - -You won't score answers from the SDK. You configure evals **in the platform** as an Eval Task, so they run on your traces and the whole team sees the same scores. The turn span already carries `raw.input`, `raw.output`, and `raw.context`, so the task has clean attributes to map to. - -In Observe, create an **Eval Task** on the `support-agent` project and pick three evals, chosen so a low score points at a *specific* component: - -- **Context Relevance** — did retrieval fetch the right policy? · map `input` → `raw.input`, `context` → `raw.context` -- **Context Adherence** — did the answer stay grounded in it? · map `output` → `raw.output`, `context` → `raw.context` -- **Completeness** — did the answer fully address the question? · map `input` → `raw.input`, `output` → `raw.output` - -Run it as **Historical** over the five turns you just sent. - - -This step only covers what to pick for this agent. The click-by-click, creating the task, choosing Historical or Continuous, setting sampling, and mapping eval inputs to span attributes, lives in the [Setup evals](/docs/observe/guides/setup-evals) how-to guide in the Observe docs. - - -**You should see** three scores per trace. Read them as a baseline (illustrative shape, your numbers will differ): - -| Turn | Context Relevance | Context Adherence | Completeness | -|---|---|---|---| -| "How long do refunds take?" | 0.91 | 0.88 | 0.86 | -| "Final sale, still refundable?" | 0.87 | **0.41** | 0.55 | -| "Look up order A-3391" | 0.93 | 0.90 | 0.84 | -| "What shipping options do you have?" | 0.90 | 0.85 | 0.83 | -| "Can you change my password for me?" | 0.88 | 0.82 | 0.80 | - - - - - -Don't guess, read the pattern. Filter the trace explorer to the low-scoring slice: - -```text -scores.context_adherence < 0.8 AND fi.span.kind = LLM -``` - -The final-sale turn surfaces, and its scores are the whole story: retrieval is fine (Context Relevance 0.87), grounding is not (Context Adherence 0.41). The right policy was in front of the model and it answered against it anyway. That points at one layer, and only one. To replay the customer's whole conversation around the bad turn, scope the table with `user.id = 'cust_42'` and open the session. - -| Eval signal | Deduction | Where the fix lives | -|---|---|---| -| Context Relevance low | wrong policy retrieved | retrieval: chunking, query, `k`, reranker, filters | -| Relevance ok, **Adherence low** | model ignores good context | **the prompt** | -| Completeness low | partial answer | prompt, or retrieve more context | - - - - - -Relevance is fine and Adherence is low, so the fix is the **prompt**, not the retriever. Constrain the agent to the retrieved policy and give it an out: - -```python -SYSTEM_PROMPT = ( - "You are a customer-support agent. Answer ONLY from the policy text returned by " - "search_help_center. Quote the specific rule you used. If the retrieved policy does " - "not cover the question, say you are not certain and offer to escalate, rather than guessing." -) - -agent = create_react_agent(llm, tools=[search_help_center, lookup_order], prompt=SYSTEM_PROMPT) -``` - -Re-run the same five turns to send fresh traces, then run the Eval Task again. This is the whole point: a code change you deduced from a score, verified by the same eval, with nothing exported and no separate tool. - -**You should see** the final-sale turn move (illustrative): - -| Turn | Context Adherence | Completeness | -|---|---|---| -| "Final sale, still refundable?" | 0.41 → **0.86** | 0.55 → **0.82** | - -The agent now says final-sale items are not refundable and cites the refund policy, instead of inventing a refund path. - - - - - -You're done when all three are true: the final-sale answer refuses a refund and cites the policy; the `support-agent` project shows the moved Context Adherence score; and you can say *why* it moved (the retriever was never the problem, the prompt was). - -A fix you can't monitor will silently regress on the next prompt tweak. Lock it in: - -- **Alert:** an [Evaluation-metric alert](/docs/observe/guides/setup-alerts) on `context_adherence`, operator *Less than*, static `0.8`. Observe pings you before a customer does. -- **Saved view:** save the `scores.context_adherence < 0.8` filter as a reusable "Ungrounded answers" view. See the [Views](/docs/observe/guides/explore-dashboard/views) guide. -- **Regression set:** the turns that scored low are your best test cases, harvest them so the next change has to clear them. - - - - - -## Troubleshooting - -| Symptom | Cause | Fix | -|---|---|---| -| No trace in Observe | `register()`/`.instrument()` ran after the agent call, or the script exited before flush | Register and instrument first; call `trace_provider.force_flush()` before exit | -| Session or user filters return nothing | The turn ran outside the `using_session` / `using_user` context | Keep the agent call inside `with using_session(...), using_user(...)` | -| Eval Task finished but no scores | The task ran Historical before the traces existed, or sampling or filters excluded them | Re-run after sending traces; widen the date range; raise the sampling rate | -| Eval reports a missing input | The turn span didn't set `raw.input` / `raw.output` / `raw.context`, or the mapping points elsewhere | Set the three attributes; map each eval input to them (Context Adherence needs `context` + `output`) | -| Alert never fires | Wrong metric or project type | Use an Evaluation-metric alert on `context_adherence`; monitors work only on `observe` projects | -| Context Relevance is the low one, not Adherence | The failure is retrieval, not the prompt | Fix the retriever (chunking, `k`, filters), not the prompt | - -## Where to go next - -You improved this agent by hand: you read the eval and changed the prompt. To close the loop *automatically*, so the agent's prompt improves without you touching it, feed the same scores into optimization in [Improve a prompt automatically](/docs/cookbook/quickstart/prompt-optimization). diff --git a/src/pages/docs/cookbook/index.mdx b/src/pages/docs/cookbook/index.mdx index d07f039a..10500ec3 100644 --- a/src/pages/docs/cookbook/index.mdx +++ b/src/pages/docs/cookbook/index.mdx @@ -111,11 +111,11 @@ description: "Practical step-by-step guides for evaluation, optimization, simula - Score a support agent with an Eval Task, find the failing turn, and fix it from the signal + Trace a support agent by session and user, score it with an Eval Task, and read the scores as insights +Build a multi-turn LangGraph support agent and instrument it with [traceAI](/docs/observe/concepts/traceai). Every conversation lands in [Observe](/docs/observe), grouped by [session](/docs/observe/concepts/sessions) and [user](/docs/observe/concepts/users). Then read three things back: the traces of what the agent did, an [Eval Task](/docs/observe/guides/setup-evals) that scores every turn, and filters that turn those scores into insight — which layer failed, on which turns, for which customer. + + +| Time | Difficulty | +|------|------------| +| 25-35 min | Intermediate | + +The loop you'll run: + + B["Read the traces"] + B --> C["Score with an Eval Task"] + C --> D["Turn scores into insight"] + D --> E["Alert & save a view"]`} /> + + +- FutureAGI account → [app.futureagi.com](https://app.futureagi.com) +- API keys: `FI_API_KEY` and `FI_SECRET_KEY` (see [Get your API keys](/docs/admin-settings)) +- An `OPENAI_API_KEY` +- Python 3.11 + + +## Install + +```bash +pip install fi-instrumentation-otel traceAI-langchain langgraph langchain-openai langchain-core +``` + +```bash +export FI_API_KEY="your-api-key" +export FI_SECRET_KEY="your-secret-key" +export OPENAI_API_KEY="your-openai-api-key" +``` + +## Tutorial + + + + + +Prove your keys and ingestion work before writing any agent code. Register against an **Observe** project, emit one span, and flush. + +```python +from fi_instrumentation import register, FITracer +from fi_instrumentation.fi_types import ProjectType + +# Reuse this trace_provider and tracer for the rest of the cookbook. +trace_provider = register( + project_type=ProjectType.OBSERVE, + project_name="support-agent", + set_global_tracer_provider=True, +) +tracer = FITracer(trace_provider.get_tracer(__name__)) + +with tracer.start_as_current_span("preflight") as span: + span.set_attribute("raw.input", "ping") + span.set_attribute("raw.output", "pong") + +trace_provider.force_flush() +print("sent preflight span") +``` + +**You should see** a `preflight` trace in **Observe → Traces → `support-agent`** within a few seconds. If it never lands, fix that here first — see [Troubleshooting](#troubleshooting). + + + + + +The agent answers from a small inline knowledge base, so the recipe runs with no external data. The `search_help_center` tool stashes what it retrieved so you can score grounding later. + +```python +from langchain_openai import ChatOpenAI, OpenAIEmbeddings +from langchain_core.vectorstores import InMemoryVectorStore +from langchain_core.tools import tool +from langgraph.prebuilt import create_react_agent + +POLICIES = [ + "Refunds: Orders can be refunded within 30 days of delivery. Items marked 'final sale' " + "are not refundable. Refunds return to the original payment method in 5-7 business days.", + "Shipping: Standard shipping takes 3-5 business days. Express takes 1-2. We ship to the US and Canada only.", + "Account: Customers can reset a password from the login page. Support cannot see or change a password.", + "Returns: To return an item, start a return from the Orders page to get a prepaid label. " + "Returns must be shipped within 14 days of approval.", +] + +store = InMemoryVectorStore.from_texts(POLICIES, OpenAIEmbeddings()) +retriever = store.as_retriever(search_kwargs={"k": 2}) + +last_context = {"text": ""} # holds the policy retrieved this turn, for scoring + +@tool +def search_help_center(query: str) -> str: + """Search the help center for relevant policy text.""" + docs = retriever.invoke(query) + last_context["text"] = "\n\n".join(d.page_content for d in docs) + return last_context["text"] + +@tool +def lookup_order(order_id: str) -> str: + """Look up the status of an order by its ID.""" + return f"Order {order_id}: delivered on 2026-06-20, standard shipping." + +SYSTEM_PROMPT = "You are a helpful customer-support agent. Use the tools to answer." + +llm = ChatOpenAI(model="gpt-4o-mini", temperature=0) +agent = create_react_agent(llm, tools=[search_help_center, lookup_order], prompt=SYSTEM_PROMPT) +``` + + +The system prompt is deliberately minimal, so the traces contain a real failure to find. Finding it is what this cookbook is about. + + + + + + +Turn on auto-instrumentation, then run the conversations. The same `LangChainInstrumentor` captures LangGraph. Wrap each turn in `using_session` (the conversation) and `using_user` (the customer), and record the three fields an eval needs: question, answer, retrieved policy. + +```python +from fi_instrumentation import using_session, using_user +from traceai_langchain import LangChainInstrumentor + +# trace_provider and tracer come from step 1 — do not register again. +LangChainInstrumentor().instrument(tracer_provider=trace_provider) + +def support_turn(session_id: str, user_id: str, question: str) -> str: + with using_session(session_id), using_user(user_id): + with tracer.start_as_current_span("support_turn") as span: + result = agent.invoke({"messages": [{"role": "user", "content": question}]}) + answer = result["messages"][-1].content + span.set_attribute("raw.input", question) + span.set_attribute("raw.output", answer) + span.set_attribute("raw.context", last_context["text"]) # retrieved policy + return answer + +# two customers, five turns between them +CONVERSATIONS = [ + ("chat_1001", "cust_42", [ + "How long do refunds take?", + "My order was a final sale, can I still get a refund?", + "Can you look up order A-3391?", + ]), + ("chat_1002", "cust_77", [ + "What shipping options do you have?", + "Can you change my password for me?", + ]), +] + +for sid, uid, questions in CONVERSATIONS: + for q in questions: + print(support_turn(sid, uid, q)) + +trace_provider.force_flush() # flush before the script exits +``` + +**You should see** two sessions in **Observe → Traces → `support-agent`**: `chat_1001` with three traces for `cust_42`, and `chat_1002` with two for `cust_77`. + + +No trace? It is almost always order or flush. `register()` and `.instrument()` must run **before** the agent call, and `force_flush()` before the process exits. + + + + + + +Before any eval runs, the traces already answer questions you'd otherwise guess at. Open the `support-agent` project: every row is one turn, with its status, model, latency, tokens, and cost. Click the final-sale turn to see the ReAct loop as a span tree — the LLM call that decides to search, the `search_help_center` call with the exact policy it returned, and the LLM call that writes the answer. + +Three things are readable before any score exists: + +- **Cost** — two LLM calls per answered question, and the token split between them +- **Latency** — the retriever and tool spans are milliseconds; the LLM spans are the wait +- **Grounding** — the tool span carries the policy text, so you can eyeball whether the answer used it + +On the final-sale turn, the retrieved policy says final-sale items are not refundable, and the answer hedges toward a refund anyway. An eval can now confirm at scale what this one trace suggests. + + + + + +Configure evals **in the platform** as an Eval Task, not from the SDK, so they run on your traces and the whole team sees the same scores. The turn span already carries `raw.input`, `raw.output`, and `raw.context` to map to. + +Create an **Eval Task** on the `support-agent` project and pick three evals, each chosen so a low score points at one component: + +- **Context Relevance** — did retrieval fetch the right policy? · `input` → `raw.input`, `context` → `raw.context` +- **Context Adherence** — did the answer stay grounded in it? · `output` → `raw.output`, `context` → `raw.context` +- **Completeness** — did the answer fully address the question? · `input` → `raw.input`, `output` → `raw.output` + +Run it as **Historical** over the five turns. + + +The click-by-click — creating the task, Historical vs Continuous, sampling, and mapping inputs to span attributes — lives in the [Setup evals](/docs/observe/guides/setup-evals) guide. + + +**You should see** three scores per trace (illustrative shape — your numbers will differ): + +| Turn | Context Relevance | Context Adherence | Completeness | +|---|---|---|---| +| "How long do refunds take?" | 0.91 | 0.88 | 0.86 | +| "Final sale, still refundable?" | 0.87 | **0.41** | 0.55 | +| "Look up order A-3391" | 0.93 | 0.90 | 0.84 | +| "What shipping options do you have?" | 0.90 | 0.85 | 0.83 | +| "Can you change my password for me?" | 0.88 | 0.82 | 0.80 | + + + + + +Filter the trace explorer to the low-scoring slice: + +```text +scores.context_adherence < 0.8 AND fi.span.kind = LLM +``` + +The final-sale turn surfaces, and its two scores split the layers: retrieval is fine (Context Relevance 0.87), grounding is not (Context Adherence 0.41). To replay the whole conversation around it, scope with `user.id = 'cust_42'` and open the session. + +Each eval's failure indicts one layer: + +| Eval signal | Insight | Where a fix lives | +|---|---|---| +| Context Relevance low | wrong policy retrieved | retrieval: chunking, query, `k`, filters | +| Relevance ok, **Adherence low** | model ignores good context | the prompt | +| Completeness low | partial answer | prompt, or retrieve more | + +**What you now know**, from one run: + +- **The prompt is the broken layer, not retrieval** — the right policy was retrieved and ignored +- **It fails on policy exceptions, not routine turns** — the plain refund question scored fine; the final-sale one didn't +- **It's one turn, not an outage** — every other turn for `cust_42` is healthy + +That's a ticket your team can act on, not "the bot sometimes gives wrong answers". + + + + + +Scores drift as prompts, models, and traffic change. Turn this check into standing monitors so the next regression pages you, not a customer: + +- **Alert** — an [Evaluation-metric alert](/docs/observe/guides/setup-alerts) on `context_adherence`, operator *Less than*, static `0.8` +- **Saved view** — save the `scores.context_adherence < 0.8` filter as an "Ungrounded answers" [view](/docs/observe/guides/explore-dashboard/views), so tomorrow's check is one click +- **Regression set** — the low-scoring turns are your best future test cases; harvest them so any fix has to clear them + + + + + +## Troubleshooting + +| Symptom | Cause | Fix | +|---|---|---| +| No trace in Observe | `register()`/`.instrument()` ran after the agent call, or the script exited before flush | Register and instrument first; `force_flush()` before exit | +| Session or user filters return nothing | The turn ran outside the `using_session` / `using_user` context | Keep the agent call inside `with using_session(...), using_user(...)` | +| Eval Task finished but no scores | It ran Historical before the traces existed, or sampling excluded them | Re-run after sending traces; widen the date range; raise sampling | +| Eval reports a missing input | The span didn't set `raw.input` / `raw.output` / `raw.context`, or the mapping points elsewhere | Set the three attributes; map each eval input to them | +| Alert never fires | Wrong metric or project type | Evaluation-metric alert on `context_adherence`; monitors work only on `observe` projects | +| Context Relevance is the low score, not Adherence | The weak spot is retrieval, not the prompt | Same method, different layer: chunking, `k`, and filters | + +## Where to go next + +You know which layer is broken and why: the prompt ignores good context on policy-exception turns. To act on that and fix the prompt systematically, continue with [Improve a prompt automatically](/docs/cookbook/quickstart/prompt-optimization). diff --git a/src/pages/docs/cookbook/quickstart/manual-tracing.mdx b/src/pages/docs/cookbook/quickstart/manual-tracing.mdx index 93132e62..9317aefb 100644 --- a/src/pages/docs/cookbook/quickstart/manual-tracing.mdx +++ b/src/pages/docs/cookbook/quickstart/manual-tracing.mdx @@ -177,7 +177,7 @@ In Tracing, filter by tags using the **Attribute** filter: select **Attribute** ![Traces filtered by tag in the Tracing dashboard](https://fi-cookbook-assets.s3.ap-south-1.amazonaws.com/quickstart/manual-tracing/step-4-tags.png) -You can combine `using_user`, `using_session`, `using_metadata`, and `using_tags` into a single `using_attributes()` call for convenience. See the [tracing reference](/docs/observe/features/manual-tracing/set-up-tracing) for details. +You can combine `using_user`, `using_session`, `using_metadata`, and `using_tags` into a single `using_attributes()` call for convenience. See the [tracing reference](/docs/sdk/tracing/set-up-tracing) for details. diff --git a/src/pages/docs/cookbook/self-hosting/docker-compose-quickstart.mdx b/src/pages/docs/cookbook/self-hosting/docker-compose-quickstart.mdx index 58c7fb85..5a368dc5 100644 --- a/src/pages/docs/cookbook/self-hosting/docker-compose-quickstart.mdx +++ b/src/pages/docs/cookbook/self-hosting/docker-compose-quickstart.mdx @@ -68,7 +68,7 @@ MAILGUN_DOMAIN=mg.your-domain.com If you don't have Mailgun, skip this. You can still create a user and set a password via the Django shell in [Step 4](#step-4). -See [Environment Variables](/docs/self-hosting/environment) for the full list of knobs. +See [Environment Variables](/docs/self-hosting/configuration/environment) for the full list of knobs. @@ -199,7 +199,7 @@ docker compose down -v Full deployment modes (full stack, dev overlay, frontend-only) - + Every secret, port, and runtime flag the stack reads diff --git a/src/pages/docs/faq.mdx b/src/pages/docs/faq.mdx index 123fbbc3..108cf801 100644 --- a/src/pages/docs/faq.mdx +++ b/src/pages/docs/faq.mdx @@ -129,7 +129,7 @@ Observe captures every LLM call, tool use, and agent decision as a trace. You ca **How do I set up alerts?** -Configure alerts to notify you about anomalies based on defined thresholds. See [Alerts & Monitors](/docs/observe/features/alerts). +Configure alerts to notify you about anomalies based on defined thresholds. See [Alerts & Monitors](/docs/observe/guides/setup-alerts). --- diff --git a/src/pages/docs/get-started/send-your-first-trace.mdx b/src/pages/docs/get-started/send-your-first-trace.mdx index 5d254402..d1f49169 100644 --- a/src/pages/docs/get-started/send-your-first-trace.mdx +++ b/src/pages/docs/get-started/send-your-first-trace.mdx @@ -144,7 +144,7 @@ Not seeing your traces? Try checking these: traceAI supports Anthropic, LangChain, LlamaIndex, and 30+ more with the same four steps - + Start scoring quality on the traces you capture diff --git a/src/pages/docs/integrations/export/pagerduty.mdx b/src/pages/docs/integrations/export/pagerduty.mdx index 64f98c28..f333ddd2 100644 --- a/src/pages/docs/integrations/export/pagerduty.mdx +++ b/src/pages/docs/integrations/export/pagerduty.mdx @@ -126,7 +126,7 @@ Your routing key may have been revoked or the PagerDuty service was deleted. Gen - + diff --git a/src/pages/docs/integrations/import/langfuse.mdx b/src/pages/docs/integrations/import/langfuse.mdx index a41473ef..ed2143a6 100644 --- a/src/pages/docs/integrations/import/langfuse.mdx +++ b/src/pages/docs/integrations/import/langfuse.mdx @@ -197,6 +197,6 @@ Large backfills can hit Langfuse's API rate limits, especially on the free tier. - + diff --git a/src/pages/docs/integrations/index.mdx b/src/pages/docs/integrations/index.mdx index c61343f7..97710ecc 100644 --- a/src/pages/docs/integrations/index.mdx +++ b/src/pages/docs/integrations/index.mdx @@ -43,7 +43,7 @@ TraceAI provides pre-built auto-instrumentation for the following frameworks and - + @@ -55,8 +55,8 @@ The Langfuse card above is for SDK-level tracing integration (sending new traces - - + + ### Other diff --git a/src/pages/docs/observe/concepts/observability-model.mdx b/src/pages/docs/observe/concepts/observability-model.mdx index 0bc7665d..a392e6ab 100644 --- a/src/pages/docs/observe/concepts/observability-model.mdx +++ b/src/pages/docs/observe/concepts/observability-model.mdx @@ -66,7 +66,7 @@ Read it bottom-up when debugging (a bad span, up to its trace, its session, its | **Trace** | One complete request, made of all the spans that share its trace ID. | Trace ID | [Traces](/docs/observe/concepts/traces) | | **Session** | A multi-turn conversation — the traces that share a session ID. | `session.id` | [Sessions](/docs/observe/concepts/sessions) | | **User** | One end user, across all their sessions and traces. | `user.id` | [Users](/docs/observe/concepts/users) | -| **Eval score** | A quality score attached to a span or trace. | Attached to span/trace | [Trace evals](/docs/observe/features/evals) | +| **Eval score** | A quality score attached to a span or trace. | Attached to span/trace | [Trace evals](/docs/observe/guides/setup-evals) | | **OpenTelemetry** | The open standard the spans are emitted in. | — | [OpenTelemetry](https://opentelemetry.io/docs/) | | **traceAI** | The SDK that produces the spans. | — | [traceAI SDK](/docs/observe/concepts/traceai) | @@ -74,7 +74,7 @@ Read it bottom-up when debugging (a bad span, up to its trace, its session, its Your app emits spans through traceAI (or OpenTelemetry directly). Each span carries a trace ID, so all spans from one request form a single trace. The backend receives them over OTLP (HTTP or gRPC) and stores them by project, and from there every Observe view runs on the same data. -To enrich the hierarchy, attach a `session.id` and a `user.id` in code, and every span inside picks them up. See [set session and user IDs](/docs/observe/features/manual-tracing/set-session-user-id) and [add attributes and metadata](/docs/observe/features/manual-tracing/add-attributes-metadata-tags). +To enrich the hierarchy, attach a `session.id` and a `user.id` in code, and every span inside picks them up. See [set session and user IDs](/docs/sdk/tracing/set-session-user-id) and [add attributes and metadata](/docs/sdk/tracing/add-attributes-metadata-tags). ## Walking the hierarchy when debugging @@ -106,7 +106,7 @@ Because the IDs link each level to the next, every climb is one click, and you n Send a trace and watch the model fill in - + Score spans and traces for quality and safety diff --git a/src/pages/docs/observe/concepts/sessions.mdx b/src/pages/docs/observe/concepts/sessions.mdx index 884459a7..c093d7b8 100644 --- a/src/pages/docs/observe/concepts/sessions.mdx +++ b/src/pages/docs/observe/concepts/sessions.mdx @@ -39,10 +39,10 @@ A conversation's problems are invisible one request at a time. Whether the assis Roll every conversation up by the person who had it - + Read, filter, and sort the Sessions view - + Attach session.id in traceAI diff --git a/src/pages/docs/observe/concepts/spans.mdx b/src/pages/docs/observe/concepts/spans.mdx index 1c3eb015..e8776757 100644 --- a/src/pages/docs/observe/concepts/spans.mdx +++ b/src/pages/docs/observe/concepts/spans.mdx @@ -29,7 +29,7 @@ Each box is a span with its own timing and attributes. The edges come straight f ## Why it matters -A response is only as strong as its weakest step, and that step is usually where things break. The trace tells you a request was slow or wrong; the span tells you which step did it and hands you the evidence: the exact prompt sent to the model, the arguments a tool received, the chunks a retriever pulled back, or the score an evaluator gave. traceAI captures these spans for you on supported frameworks, and where it can't reach, you can [add your own](/docs/traceai/manual-instrumentation/create-tool-spans) so no part of your pipeline stays a black box. +A response is only as strong as its weakest step, and that step is usually where things break. The trace tells you a request was slow or wrong; the span tells you which step did it and hands you the evidence: the exact prompt sent to the model, the arguments a tool received, the chunks a retriever pulled back, or the score an evaluator gave. traceAI captures these spans for you on supported frameworks, and where it can't reach, you can [add your own](/docs/sdk/tracing/create-tool-spans) so no part of your pipeline stays a black box. ## Span types @@ -62,7 +62,7 @@ Every span carries key-value attributes. Some come straight from [OpenTelemetry] The full request that spans are grouped into - + Add custom spans where auto-instrumentation stops diff --git a/src/pages/docs/observe/concepts/traceai.mdx b/src/pages/docs/observe/concepts/traceai.mdx index edf6ac68..f85bcfbb 100644 --- a/src/pages/docs/observe/concepts/traceai.mdx +++ b/src/pages/docs/observe/concepts/traceai.mdx @@ -1,158 +1,45 @@ --- -title: "traceAI SDK" -description: "traceAI is FutureAGI's open-source instrumentation SDK on OpenTelemetry — it captures LLM, tool, and retrieval calls as standardized spans that Observe reads as traces." -slug: "traceai" -page_type: "concept" -diataxis: "explanation" -products: ["Observe"] -concept_family: "tracing" -concept_level: "foundational" -primary_question: "What is the traceAI SDK?" -direct_answer: "traceAI is FutureAGI's open-source instrumentation SDK on OpenTelemetry. It captures model, tool, and retrieval calls as standardized spans, and Observe is the product that reads them as traces." -audience: ["engineer"] -difficulty: "beginner" -status: "review" -owner: "observability" -reviewers: ["observability-eng"] -last_tested: "2026-06-18" -last_diagram_reviewed: "2026-05-25" -schema_type: "TechArticle" -seo: - title: "The traceAI SDK" - description: "traceAI is FutureAGI's open-source instrumentation SDK on OpenTelemetry. It captures LLM, tool, and retrieval calls as standardized spans that Observe reads as traces." - primary_keyword: "what is traceai instrumentation" - direct_answer: true -geo: - answer_target: "What is the traceAI SDK and how does it relate to Observe?" - llm_summary: "traceAI is FutureAGI's open-source instrumentation SDK, built on OpenTelemetry. It captures model, tool, and retrieval calls as standardized spans across frameworks; Observe is the product that reads those spans as traces." -canonical: "/docs/observe/concepts/traceai" -related: - - "https://opentelemetry.io/docs/" - - "/docs/observe/concepts/spans" - - "/docs/traceai/manual-instrumentation/set-up-tracing" - - "/docs/traceai/auto" +title: "traceAI" +description: "Turning your framework's model, tool, and retrieval calls into standardized spans Observe reads as traces." --- -## About +## traceAI is the instrumentation SDK -traceAI is the instrumentation SDK; Observe is the product that reads its traces. traceAI is FutureAGI's open-source instrumentation SDK, built on [OpenTelemetry](https://opentelemetry.io/docs/). It's a set of conventions and per-framework instrumentors that capture what your AI app does — model calls, tool calls, retrievals, agent steps — and map them to standardized [span](/docs/observe/concepts/spans) attributes. Add the instrumentor for your framework, and those calls become traces in Observe without hand-writing spans. traceAI is natively supported by FutureAGI but emits standard OTel, so it works with any OTel-compatible backend too. - ---- - -## Why it matters - -Raw OpenTelemetry knows nothing about LLMs — it has no concept of a prompt, a completion, token cost, or a tool call. traceAI fills that gap: it turns framework calls into *LLM-shaped* spans with consistent keys, so a LangChain trace and an OpenAI trace look the same in Observe and are queryable the same way. That standardization is what makes filtering, evals, and dashboards work across different stacks. - ---- +**traceAI** is Future AGI's open-source instrumentation SDK, built on [OpenTelemetry](https://opentelemetry.io/docs/). It's a set of conventions and per-framework instrumentors that capture what your AI app does (model calls, tool calls, retrievals, agent steps) and map them to standardized [span](/docs/observe/concepts/spans) attributes. Add the instrumentor for your framework, and those calls become traces in [Observe](/docs/observe) with no spans written by hand. traceAI runs inside your application and produces the spans; Observe is the product that reads them. They meet only at the span, a standard OpenTelemetry span on the wire, so the same output also works with any OTel-compatible backend. ## Mental model -traceAI is the adapter between your framework and OpenTelemetry: the instrumentor wraps the framework, produces standardized spans, and hands them to the OTel pipeline that exports to FutureAGI. +traceAI is the adapter between your framework and OpenTelemetry. The instrumentor wraps the framework, produces standardized spans, and hands them to the OTel pipeline that exports to Future AGI. B["traceAI instrumentor"] B --> C["Standardized OTel spans"] - C --> D["FutureAGI Observe"]`} /> - -You pick the instrumentor that matches your framework; the rest of the pipeline is the same OTel flow for everyone. - ---- - -## SDK vs product, auto vs manual - -Two distinctions explain most of how traceAI is used. - -**traceAI is the SDK; Observe is the product.** traceAI runs inside your application and produces spans. [Observe](/docs/observe) runs in FutureAGI and reads them — searching, replaying, scoring, and alerting on the traces those spans form. One emits the data; the other consumes it. They meet only at the span: a standard [OpenTelemetry](https://opentelemetry.io/docs/) span on the wire. - -**Auto and manual instrumentation are two ways to produce those spans, and they coexist.** Auto-instrumentation is a per-framework instrumentor that wraps a library — install `traceAI-openai`, call `.instrument()`, and every OpenAI call becomes a span with no span code in your app. Manual instrumentation is for the parts no instrumentor reaches: your own business functions, custom retrieval, glue logic. You wrap those as [tool spans](/docs/traceai/manual-instrumentation/create-tool-spans) yourself. Most real apps use both — auto for the framework calls, manual for the code between them — and both feed the same provider, so they nest into one trace. - -That shared provider is what `register()` sets up. Conceptually it does one thing: it builds the OpenTelemetry `TracerProvider` — the exporter pointed at FutureAGI plus a batched span processor — and makes it the active provider. After that call, both the auto-instrumentors and any manual spans attach to it automatically. Nothing reaches Observe until `register()` has run, because before it there is no exporter to ship spans to. - ---- - -## When to use - -- You want LLM/agent calls traced without writing spans by hand. -- You use a supported framework (OpenAI, LangChain, LlamaIndex, CrewAI, …). -- You want consistent, queryable span attributes across different SDKs. -- You want instrumentation that stays portable across OTel backends. + C --> D["Future AGI Observe"]`} /> ---- - -## When not to use traceAI - -- **For non-LLM work that has no LLM meaning to capture.** A plain database query or HTTP handler doesn't benefit from traceAI's LLM conventions. Trace it with raw [OpenTelemetry](https://opentelemetry.io/docs/) instead; it still lands in the same trace tree. -- **As a stand-in for the backend.** traceAI only produces spans — it doesn't store, search, or display them. If you need to read traces, that's [Observe](/docs/observe), not the SDK. -- **For a framework with no instrumentor, expecting auto-capture.** Each instrumentor wraps one specific framework. If yours isn't in the [catalog](/docs/traceai/auto), auto-instrumentation won't see it — reach for [manual spans](/docs/traceai/manual-instrumentation/create-tool-spans) rather than the wrong instrumentor. - ---- - -## What it isn't - -- **traceAI is not a backend.** It produces spans; FutureAGI stores and displays them. -- **traceAI is not a replacement for OpenTelemetry.** It's complementary — conventions and instrumentors *on top of* OTel. See [OpenTelemetry](https://opentelemetry.io/docs/). -- **traceAI is not only for FutureAGI.** It emits standard OTel and works with any compatible backend. - ---- - -## How FutureAGI represents traceAI +You pick the instrumentor that matches your framework, and the rest of the pipeline is the same OTel flow for everyone. -traceAI ships as the core `fi-instrumentation-otel` package plus a per-framework instrumentor you install alongside it. A sample of the Python instrumentors: +## Auto and manual instrumentation -| Package | Instruments | -|---|---| -| `traceAI-openai` | OpenAI | -| `traceAI-anthropic` | Anthropic | -| `traceAI-langchain` | LangChain | -| `traceAI-llamaindex` | LlamaIndex | -| `traceAI-crewai` | CrewAI | -| `traceAI-bedrock` | AWS Bedrock | -| `traceAI-litellm` | LiteLLM | -| `traceAI-google-adk` | Google ADK | -| `traceAI-dspy` | DSPy | -| `traceAI-haystack` | Haystack | +There are two ways to produce spans, and real apps use both. Auto-instrumentation is a per-framework instrumentor that wraps a library: install the one for your framework (`traceAI-openai`, `traceAI-langchain`, and so on), call `.instrument()`, and every framework call becomes a span with no span code in your app. Manual instrumentation covers the parts no instrumentor reaches, like your own business functions, custom retrieval, or glue logic, which you wrap as [tool spans](/docs/sdk/tracing/create-tool-spans) yourself. -See the full, current list in the [Auto Instrumentation catalog](/docs/traceai/auto). To wire one up, see [Set up tracing](/docs/traceai/manual-instrumentation/set-up-tracing). +Auto and manual spans feed the same provider, so they nest into one trace. That shared provider is what `register()` sets up: it builds the OpenTelemetry tracer provider, points the exporter at Future AGI, and makes it active. Nothing reaches Observe until `register()` has run, because before it there's no exporter to ship spans to. Plain non-LLM work, like a database query, needs none of this: trace it with raw OpenTelemetry and it still lands in the same trace tree. -Install the core package and an instrumentor, register a tracer provider, then instrument the framework with it: - - -```python -# pip install fi-instrumentation-otel traceAI-openai -from fi_instrumentation import register, Transport -from fi_instrumentation.fi_types import ProjectType -from traceai_openai import OpenAIInstrumentor - -tp = register(project_type=ProjectType.OBSERVE, project_name="my-app", transport=Transport.GRPC) -OpenAIInstrumentor().instrument(tracer_provider=tp) -``` - - ---- - -## Common mistakes - -- **Calling `.instrument()` after the client is created → no spans.** Run `OpenAIInstrumentor().instrument(...)` before you construct the framework client, or its calls aren't wrapped and nothing is traced. -- **Registering with the wrong `project_type`.** Use `ProjectType.OBSERVE` for production tracing; a mismatched project type sends spans somewhere you won't find them in Observe. -- **Installing the wrong instrumentor for your framework.** Each instrumentor wraps one framework. A LangChain app needs `traceAI-langchain`; `traceAI-openai` won't capture it. +## Why it matters ---- +Raw OpenTelemetry knows nothing about LLMs. It has no concept of a prompt, a completion, token cost, or a tool call. traceAI fills that gap. It turns framework calls into LLM-shaped spans with consistent keys, so a LangChain trace and an OpenAI trace look the same in Observe and are queryable the same way. That standardization is what lets filters, evals, and dashboards work across different stacks instead of breaking every time you change frameworks. -## Next steps +## Keep exploring - - - The standard traceAI is built on. - - - What traceAI produces. + + + What traceAI produces - - Install an instrumentor and start capturing. + + Install an instrumentor and start capturing - - Every supported framework. + + Every supported framework diff --git a/src/pages/docs/observe/concepts/traces.mdx b/src/pages/docs/observe/concepts/traces.mdx index d5cd1cfc..381e09ff 100644 --- a/src/pages/docs/observe/concepts/traces.mdx +++ b/src/pages/docs/observe/concepts/traces.mdx @@ -22,7 +22,7 @@ Read top to bottom, the tree is the exact path the request took, so when an answ ## What a trace isn't -- **Not a session.** A session bundles many traces from one conversation or user. A trace is just one request inside it. See [Sessions and users](/docs/observe/features/session) +- **Not a session.** A session bundles many traces from one conversation or user. A trace is just one request inside it. See [Sessions and users](/docs/observe/concepts/sessions) - **Not a log line.** Logs are flat text events. A trace is a timed, structured tree with inputs, outputs, and cost at every step ## Why it matters @@ -35,7 +35,7 @@ Without traces, a wrong or slow answer is a dead end. You see the output but not Group multiple traces into one conversation or customer - + Instrument your app so it emits traces diff --git a/src/pages/docs/observe/concepts/users.mdx b/src/pages/docs/observe/concepts/users.mdx index ccb862c1..c7414e55 100644 --- a/src/pages/docs/observe/concepts/users.mdx +++ b/src/pages/docs/observe/concepts/users.mdx @@ -21,7 +21,7 @@ Say one customer is `user.id="cust_42"`. Every request they make carries that ID ## When to use -Reach for a user when the unit is a person, not a chat: a customer reports a bug and you want their entire history, a cost spike traces back to one heavy customer, you are tracking who stuck around and who dropped off (first seen, last seen, session counts), or quality is slipping for a segment and you want [eval](/docs/observe/features/evals) pass-rate per user. +Reach for a user when the unit is a person, not a chat: a customer reports a bug and you want their entire history, a cost spike traces back to one heavy customer, you are tracking who stuck around and who dropped off (first seen, last seen, session counts), or quality is slipping for a segment and you want [eval](/docs/observe/guides/setup-evals) pass-rate per user. When the grain is wrong, reach elsewhere: @@ -42,10 +42,10 @@ A single trace or session shows one moment; many of the questions that matter ar Group a person's traces into individual conversations - + Read, filter, and sort the Users view - + Attach user.id in traceAI diff --git a/src/pages/docs/observe/concepts/voice-observability.mdx b/src/pages/docs/observe/concepts/voice-observability.mdx index a2f2571e..a008b9fb 100644 --- a/src/pages/docs/observe/concepts/voice-observability.mdx +++ b/src/pages/docs/observe/concepts/voice-observability.mdx @@ -5,7 +5,7 @@ description: "Turning every voice call into a trace you can debug and score." ## A voice call is a trace -**Voice observability** captures each voice call as a [trace](/docs/observe/concepts/traces), the same tree of [spans](/docs/observe/concepts/spans) you get from a text app. One call becomes one trace, and each back-and-forth turn is a span inside it. The call carries the transcript, the recording, and its duration, turn count, and cost. A spoken conversation lands in the same place as every other request, ready for the same [evals](/docs/observe/features/evals), alerts, and filters. +**Voice observability** captures each voice call as a [trace](/docs/observe/concepts/traces), the same tree of [spans](/docs/observe/concepts/spans) you get from a text app. One call becomes one trace, and each back-and-forth turn is a span inside it. The call carries the transcript, the recording, and its duration, turn count, and cost. A spoken conversation lands in the same place as every other request, ready for the same [evals](/docs/observe/guides/setup-evals), alerts, and filters. ## Inside a voice call @@ -32,7 +32,7 @@ A voice call reaches Observe by one of two paths. Whichever it takes, it lands a | **Managed ingestion** | Hosted agents on Vapi or Retell | Observe pulls the provider's call logs | No code: connect the provider and turn observability on | | **Auto-instrumentation** | Apps built on LiveKit or Pipecat | Your app emits a span per turn through [traceAI](/docs/observe/concepts/traceai) | A few lines of traceAI setup | -For the managed-ingestion setup, see [Voice observability](/docs/observe/features/voice). +For the managed-ingestion setup, see [Voice observability](/docs/observe/concepts/voice-observability). ## Debugging a call @@ -54,7 +54,7 @@ Voice failures are the ones you hear about from a customer, not a log. A spoken ## Keep exploring - + Score voice conversations for quality and safety diff --git a/src/pages/docs/observe/features/quickstart.mdx b/src/pages/docs/observe/features/quickstart.mdx index f810737d..e3362531 100644 --- a/src/pages/docs/observe/features/quickstart.mdx +++ b/src/pages/docs/observe/features/quickstart.mdx @@ -79,7 +79,7 @@ This is how you connect your application to Future AGI so LLM calls are captured Use one of two options: - **Auto Instrumentor**: For supported frameworks (e.g. OpenAI). Use Future AGI's [Auto Instrumentation](/docs/tracing/auto); recommended for most apps. - - **Manual tracing**: For custom spans, use [OpenTelemetry](/docs/tracing/concepts/otel). [Learn more →](/docs/observe/features/manual-tracing/set-up-tracing) + - **Manual tracing**: For custom spans, use [OpenTelemetry](/docs/tracing/concepts/otel). [Learn more →](/docs/sdk/tracing/set-up-tracing) Example with the OpenAI instrumentor: install the package, instrument with your trace provider, then use the OpenAI client as usual. Traces appear in your [Observe dashboard](https://app.futureagi.com/dashboard/projects/observe). diff --git a/src/pages/docs/observe/features/session.mdx b/src/pages/docs/observe/features/session.mdx index f7b8edb7..9cde25a0 100644 --- a/src/pages/docs/observe/features/session.mdx +++ b/src/pages/docs/observe/features/session.mdx @@ -3,13 +3,13 @@ title: "Explore sessions & users" description: "Read, filter, and sort the Sessions and Users views in Observe." --- -Once your spans carry `session.id` and `user.id`, Observe groups them into two views: **Sessions**, one row per conversation, and **Users**, one row per end user. This page is how to read, filter, and sort them. To attach the IDs in the first place, see [Set session and user IDs](/docs/traceai/manual-instrumentation/set-session-user-id); for the concepts, see [Sessions](/docs/observe/concepts/sessions) and [Users](/docs/observe/concepts/users). +Once your spans carry `session.id` and `user.id`, Observe groups them into two views: **Sessions**, one row per conversation, and **Users**, one row per end user. This page is how to read, filter, and sort them. To attach the IDs in the first place, see [Set session and user IDs](/docs/sdk/tracing/set-session-user-id); for the concepts, see [Sessions](/docs/observe/concepts/sessions) and [Users](/docs/observe/concepts/users). ## The Sessions view Open the project and switch to the **Sessions** tab. Each row is one conversation. -Observe Sessions tab listing conversations with first and last message, duration, total cost, and trace count +Observe Sessions tab for the support-agent project listing conversations chat_1001 and chat_1002 with first and last message, duration, total cost, and trace count *One row per conversation. Sort by total cost or total traces to find the longest or most expensive sessions.* The columns roll each conversation up at a glance: @@ -25,8 +25,8 @@ The columns roll each conversation up at a glance: Open a session for its detail view, the traces in order, each with its eval scores and annotations. From there you open any trace for the full span tree. -Session detail view showing the traces of one conversation in order with per-turn timing and cost -*A session opened: its traces in order, with per-turn timing and cost.* +Session detail with the Session History conversation and a Trace Details panel showing trace ID, evaluations, and metadata +*A session opened: the conversation turn by turn, with each turn's trace, evals, and metadata.* Narrow the list with the filter bar, by `session.id`, metadata, or any span attribute (see the [filter syntax reference](/docs/observe/reference/filters)), and scope it to a time window with the date-range picker, which recomputes the column metrics for the window. For voice and other replayable sessions, configure session replay to step back through a conversation as it happened. @@ -34,7 +34,7 @@ Narrow the list with the filter bar, by `session.id`, metadata, or any span attr Switch to the **Users** view. Each row is one end user. -Observe Users view listing end users with User ID, First Active, Last Active, number of traces, and number of sessions columns +Observe Users view for the support-agent project listing end users cust_42 and cust_77 with User ID, First Active, Last Active, number of traces, and number of sessions columns *One row per user, with trace and session counts rolled up. Sort by trace or session count to find your most active users.* The columns roll each user up: @@ -49,8 +49,11 @@ The columns roll each user up: Open a user for their detail view, where cost, evals, and guardrail results break down per session and trace, across a Traces tab and a Sessions tab. -User detail view with summary metrics, a Traces tab, and a Sessions tab for one end user -*User detail: a Traces tab and a Sessions tab for one end user.* +User detail Traces tab listing every trace for user cust_77 with input, output, status, and latency +*The Traces tab: every trace attributed to cust_77.* + +User detail Sessions tab listing every conversation for user cust_77 with first and last message, duration, cost, and trace count +*The Sessions tab: every conversation cust_77 had.* Filter and scope the same way as sessions, by `user.id`, metadata, or any span attribute, and by date range. @@ -63,7 +66,7 @@ Filter and scope the same way as sessions, by `user.id`, metadata, or any span a | One person split across users | A different `user.id` was used on some requests | Reuse one stable string for that person | | Row exists but no metrics | Spans carried the ID but no cost or token attributes | Confirm the LLM spans are auto-instrumented | -For every way to attach the IDs, see [Set session and user IDs](/docs/traceai/manual-instrumentation/set-session-user-id). +For every way to attach the IDs, see [Set session and user IDs](/docs/sdk/tracing/set-session-user-id). ## Related @@ -74,7 +77,7 @@ For every way to attach the IDs, see [Set session and user IDs](/docs/traceai/ma What a user is and when to use one - + Attach session.id and user.id in traceAI diff --git a/src/pages/docs/observe/guides/explore-dashboard/display-options.mdx b/src/pages/docs/observe/guides/explore-dashboard/display-options.mdx index 95bbda19..3641cb0e 100644 --- a/src/pages/docs/observe/guides/explore-dashboard/display-options.mdx +++ b/src/pages/docs/observe/guides/explore-dashboard/display-options.mdx @@ -3,10 +3,70 @@ title: "Display options" description: "Choose which columns show, how rows are grouped, and how the trace table is laid out." --- -Display options control how the trace table looks: which columns are visible, how they are ordered, and how rows are grouped. Tune them on the `self-improving-agent` project from the [quickstart](/docs/observe/quickstart) so the table shows what you care about. +Display options control how the trace explorer looks: which primary view sits above the table, which columns are visible and in what order, and how rows are grouped. This guide tunes them on the `support-agent` project from the [Observing a LangGraph agent](/docs/cookbook/observe-langgraph-agent-and-obtain-insights) cookbook. -## In this guide +## Open the Display menu -- Show, hide, and reorder columns -- Group rows and adjust density -- Save your layout as part of a view +Click **Display** at the top right of the trace explorer. One menu holds the primary view, row height, columns, metrics, and grouping. + +The Display menu open over the trace table, with view tabs on top and sections for rows, columns, metrics, group, and settings +*The Display menu: view switcher on top, then rows, columns, metrics, and grouping* + +## Switch the primary view + +The tabs at the top of the menu swap the graph above the table between three views of the same traces: + +- **Graph View**: the default latency and traffic time series +- **Agent Graph**: your agent's structure as a node graph +- **Agent Path**: span flow as a Sankey diagram, sized by span count + +Display menu with the Graph View, Agent Graph, and Agent Path tabs at the top +*Switch views from the tabs at the top of the menu* + +Agent Graph draws the nodes the way your agent runs them: `support_turn` into the LangGraph nodes, the tools, and the LLM calls. + +Agent Graph view showing support_turn branching into LangGraph, agent, tools, ChatOpenAI, and VectorStoreRetriever nodes +*Agent Graph: the support agent's structure as a node graph* + +Agent Path shows the same run as flow, with each node's span count, so you can see where calls concentrate. + +Agent Path Sankey diagram showing span flow from support_turn through LangGraph, agent, tools, and the LLM and retriever spans with span counts +*Agent Path: span flow sized by how many spans ran at each step* + +## Choose your columns + +Under **Columns**, **View columns** shows or hides any column, **Autosize columns** fits them to their content, and **Add custom columns** surfaces a span attribute as its own column. You can also reorder the columns so the fields you care about come first. + + @@ -35,7 +35,7 @@ A few short pages give you the whole mental model behind Observe. Read these and Follow a full conversation, or one customer across sessions - + The open library that sends your traces to Observe @@ -45,10 +45,10 @@ A few short pages give you the whole mental model behind Observe. Read these and Every other feature in Observe is just a different lens on the traces you capture: - + Attach quality scores to whole traces or single spans - + Get told the moment a metric slips diff --git a/src/pages/docs/observe/quickstart.mdx b/src/pages/docs/observe/quickstart.mdx index 9c83118d..a35eb85e 100644 --- a/src/pages/docs/observe/quickstart.mdx +++ b/src/pages/docs/observe/quickstart.mdx @@ -121,7 +121,7 @@ Pin the packages to the version you test against, so a later release cannot chan That row is a [trace](/docs/observe/concepts/traces), the full record of one request. Because this example made a single OpenAI call, the trace holds one [span](/docs/observe/concepts/spans): the `llm` operation, carrying the model, the prompt and completion, the token counts, and the cost. -The same four steps instrument 30+ frameworks. Swap the instrumentor for your stack and the flow is identical, see [all framework integrations](/docs/traceai/auto). +The same four steps instrument 30+ frameworks. Swap the instrumentor for your stack and the flow is identical, see [all framework integrations](/docs/tracing/auto). ## Not seeing your trace? @@ -138,10 +138,10 @@ The same four steps instrument 30+ frameworks. Swap the instrumentor for your st One step inside a trace: a model call, a tool call, a retrieval - + Attach quality scores to your production traces - + Get told the moment a metric slips diff --git a/src/pages/docs/observe/reference/export-formats.mdx b/src/pages/docs/observe/reference/export-formats.mdx index 10cf14e1..9722fdc8 100644 --- a/src/pages/docs/observe/reference/export-formats.mdx +++ b/src/pages/docs/observe/reference/export-formats.mdx @@ -21,8 +21,8 @@ geo: llm_summary: "Observe exports the current trace-explorer view (filters and time range) to CSV from the download icon. Inbound, the traceAI SDK sends spans over OTLP to FutureAGI — set FI_BASE_URL (HTTP) or FI_GRPC_URL (gRPC), defaulting to FutureAGI cloud, or point them at your own collector when self-hosted." canonical: "/docs/observe/reference/export-formats" related: - - "/docs/observe/features/llm-tracing" - - "/docs/observe/features/manual-tracing/set-up-tracing" + - "/docs/observe/guides/explore-dashboard" + - "/docs/sdk/tracing/set-up-tracing" - "https://opentelemetry.io/docs/" --- @@ -32,7 +32,7 @@ There are two directions for trace data: **out of** Observe (exporting what you' ## Export from the trace explorer -The download icon in the [trace explorer](/docs/observe/features/llm-tracing) header exports the **current view** — the traces that match your active filters and time range. +The download icon in the [trace explorer](/docs/observe/guides/explore-dashboard) header exports the **current view** — the traces that match your active filters and time range. **CSV is the only export format.** There is no JSON or Parquet download from the trace explorer. @@ -60,7 +60,7 @@ When unset, both variables default to the FutureAGI cloud collector, so a cloud - **Cloud:** leave the defaults; set `FI_API_KEY` and `FI_SECRET_KEY`. - **Self-hosted:** point `FI_BASE_URL` / `FI_GRPC_URL` at your own collector host so spans stay in your network. -Choose the transport with `transport=Transport.HTTP` (default) or `Transport.GRPC` in `register()`. See [Set up tracing](/docs/observe/features/manual-tracing/set-up-tracing). +Choose the transport with `transport=Transport.HTTP` (default) or `Transport.GRPC` in `register()`. See [Set up tracing](/docs/sdk/tracing/set-up-tracing). ### Endpoint contract @@ -73,20 +73,20 @@ The ingestion endpoint is an **OTLP/traces** receiver — you don't call it dire | Auth | `FI_API_KEY` + `FI_SECRET_KEY` from the [keys page](https://app.futureagi.com/dashboard/keys), sent by the SDK on every export. Keys are workspace-scoped. | | Success | The exporter batches spans and sends them in the background; a successful export returns no payload. Spans appear in Observe within seconds. | | Errors | A `401` means the keys are wrong for this workspace; a `4xx` means a malformed/oversized batch; transient `5xx`/network errors are retried by the batch exporter. | -| Limits | Spans are sent by the **batch** span processor on an interval, so a short-lived process must call `trace_provider.force_flush()` before exit or the last batch is lost. Very large payloads (huge prompts/outputs) can be dropped — [mask or trim](/docs/observe/features/manual-tracing/mask-span-attributes) them at the SDK. | +| Limits | Spans are sent by the **batch** span processor on an interval, so a short-lived process must call `trace_provider.force_flush()` before exit or the last batch is lost. Very large payloads (huge prompts/outputs) can be dropped — [mask or trim](/docs/sdk/tracing/mask-span-attributes) them at the SDK. | | Versioning | Pin `fi-instrumentation-otel` and each instrumentor to a tested version so a release can't change span shape under you; the wire format follows the OTLP version the SDK ships. | - Span input and output can carry customer data before they leave your process. Redact at the SDK with `TraceConfig` or the `FI_HIDE_*` variables — see [Mask span attributes](/docs/observe/features/manual-tracing/mask-span-attributes). + Span input and output can carry customer data before they leave your process. Redact at the SDK with `TraceConfig` or the `FI_HIDE_*` variables — see [Mask span attributes](/docs/sdk/tracing/mask-span-attributes). ## Related - + Filter, then export the current view. - + Configure the OTLP endpoint and transport. diff --git a/src/pages/docs/observe/reference/filters.mdx b/src/pages/docs/observe/reference/filters.mdx index b5a5e678..cf194845 100644 --- a/src/pages/docs/observe/reference/filters.mdx +++ b/src/pages/docs/observe/reference/filters.mdx @@ -3,7 +3,7 @@ title: "Filters" description: "How to filter traces in Observe: the filter modes, every filterable property, the metrics you can filter and aggregate on, and ready-to-paste queries." --- -The **Filter** panel in the [trace explorer](/docs/observe/features/llm-tracing) narrows which traces are shown. It offers three modes: plain-language AI search, a Basic property/condition/value builder, and a Query expression for power users. This page lists the modes, every property you can filter on, the metrics you can filter and aggregate by, and a set of ready-to-paste queries. +The **Filter** panel in the [trace explorer](/docs/observe/guides/explore-dashboard) narrows which traces are shown. It offers three modes: plain-language AI search, a Basic property/condition/value builder, and a Query expression for power users. This page lists the modes, every property you can filter on, the metrics you can filter and aggregate by, and a set of ready-to-paste queries. ## Filter modes @@ -70,7 +70,7 @@ tag.tags contains needs-review # traces carrying a tag ## Metrics -Alongside the properties above, these are the metrics Observe computes from your spans: the values you sort the trace table by and aggregate on a [dashboard](/docs/observe/features/dashboard) widget. Nothing is precomputed, each is derived from the spans that match your filters and time window. +Alongside the properties above, these are the metrics Observe computes from your spans: the values you sort the trace table by and aggregate on a [dashboard](/docs/observe/guides/explore-dashboard) widget. Nothing is precomputed, each is derived from the spans that match your filters and time window. | Metric | Unit | What it measures | |---|---|---| diff --git a/src/pages/docs/observe/troubleshooting/alerts-did-not-fire.mdx b/src/pages/docs/observe/troubleshooting/alerts-did-not-fire.mdx index f111020f..181cf666 100644 --- a/src/pages/docs/observe/troubleshooting/alerts-did-not-fire.mdx +++ b/src/pages/docs/observe/troubleshooting/alerts-did-not-fire.mdx @@ -25,8 +25,8 @@ geo: llm_summary: "When a metric crosses a threshold but no alert arrives, check timing first — a monitor only evaluates on its alert_frequency (min 5, default 60 min) — then whether it's muted, whether the threshold operator and value match the breach, whether a percentage-change baseline has enough history, and whether the email or Slack notification channel is configured correctly." canonical: "/docs/observe/troubleshooting/alerts-did-not-fire" related: - - "/docs/observe/features/alerts" - - "/docs/observe/features/dashboard" + - "/docs/observe/guides/setup-alerts" + - "/docs/observe/guides/explore-dashboard" - "/docs/observe/reference/filters" --- @@ -39,9 +39,6 @@ A metric crossed what you thought was the threshold, but no email or Slack arriv - Alerts used to arrive and stopped. - The alert log shows nothing for the period you expected. -Observe alerts list showing an Active 'High latency (p95 > 2s)' monitor with Last Triggered '-' and 0 triggers -*An alert configured but never fired: Status is Active, yet Last Triggered shows '-' and the trigger count is 0. That points to schedule timing, a mute, or a threshold set differently than expected — not a broken metric.* - --- ## Quick checks @@ -84,10 +81,10 @@ If the monitor still won't fire on a confirmed breach, contact support@futureagi ## Next steps - + How monitors and thresholds are configured. - + Confirm the metric trend the alert watches. diff --git a/src/pages/docs/observe/troubleshooting/dashboard-numbers-look-wrong.mdx b/src/pages/docs/observe/troubleshooting/dashboard-numbers-look-wrong.mdx index 53a60b07..6ea12424 100644 --- a/src/pages/docs/observe/troubleshooting/dashboard-numbers-look-wrong.mdx +++ b/src/pages/docs/observe/troubleshooting/dashboard-numbers-look-wrong.mdx @@ -25,8 +25,8 @@ geo: llm_summary: "When a dashboard widget shows an unexpected number, the data is almost always right and the query is reading it differently than assumed. Check the time range, granularity, aggregation, and filters first — plus eval sampling and timezone — and cross-check one value against the trace explorer for the same window." canonical: "/docs/observe/troubleshooting/dashboard-numbers-look-wrong" related: - - "/docs/observe/features/dashboard" - - "/docs/observe/features/llm-tracing" + - "/docs/observe/guides/explore-dashboard" + - "/docs/observe/guides/explore-dashboard" - "/docs/observe/reference/filters" --- @@ -39,9 +39,6 @@ A widget shows a number that doesn't match what you expected — cost too low, l - Two widgets that "should" match don't. - A number changed when you only changed the time range or granularity. -Observe System Metrics dashboard with latency, tokens, traffic, and cost charts over a 30-day range -*The System Metrics view. When a number here looks off, the data is usually right and the query is reading it differently — check the time range, granularity, aggregation, and filters before suspecting the traces.* - --- ## Quick checks @@ -63,7 +60,7 @@ A widget shows a number that doesn't match what you expected — cost too low, l ## Diagnostic checks -Open the widget editor and read its **time range, granularity, aggregation, group-by, and filters**. Then cross-check one value against the [trace explorer](/docs/observe/features/llm-tracing) for the exact same window: +Open the widget editor and read its **time range, granularity, aggregation, group-by, and filters**. Then cross-check one value against the [trace explorer](/docs/observe/guides/explore-dashboard) for the exact same window: - Apply the same time range and filters in the trace explorer. - Count the matching traces (or read the latency/cost column) and compare to the widget. @@ -85,10 +82,10 @@ If a value still can't be reconciled with the trace list for the same window, co ## Next steps - + How widgets are configured. - + Cross-check a number against the raw traces. diff --git a/src/pages/docs/observe/troubleshooting/missing-attributes.mdx b/src/pages/docs/observe/troubleshooting/missing-attributes.mdx index d9aa3ba8..51eac428 100644 --- a/src/pages/docs/observe/troubleshooting/missing-attributes.mdx +++ b/src/pages/docs/observe/troubleshooting/missing-attributes.mdx @@ -25,9 +25,9 @@ geo: llm_summary: "If a trace shows but spans or fields like input/output are missing, check redaction first (FI_HIDE_INPUTS/OUTPUTS or TraceConfig masking makes blanks expected), then confirm the framework's instrumentor is attached, that custom attributes were set on the active span before it closed, and that you're filtering on indexed semantic-convention keys with supported value types." canonical: "/docs/observe/troubleshooting/missing-attributes" related: - - "/docs/observe/features/llm-tracing" - - "/docs/observe/features/manual-tracing/add-attributes-metadata-tags" - - "/docs/observe/features/manual-tracing/mask-span-attributes" + - "/docs/observe/guides/explore-dashboard" + - "/docs/sdk/tracing/add-attributes-metadata-tags" + - "/docs/sdk/tracing/mask-span-attributes" - "/docs/observe/concepts/spans" --- @@ -40,9 +40,6 @@ The trace shows up, but it's incomplete — a nested span is missing, or fields - A framework's child spans (e.g. nested LangGraph nodes) don't appear. - A custom attribute you set isn't on the span, or you can't filter by it. -Observe trace explorer showing an empty trace list — 'No traces found' — with the attribute columns present but unpopulated -*When spans or attributes aren't arriving, the trace list reads empty like this — the columns exist but nothing fills them. Confirm the instrumentor is attached and that redaction isn't hiding fields before assuming data was dropped.* - --- ## Quick checks @@ -50,16 +47,16 @@ The trace shows up, but it's incomplete — a nested span is missing, or fields - Redaction is **off** for the fields you expect to see (`FI_HIDE_INPUTS` / `FI_HIDE_OUTPUTS` and `TraceConfig` masking). - The framework's instrumentor is installed and `instrument()` ran against the same tracer provider. - Custom attributes are set while the span is **still active**, before its `with` block closes. -- Anything you filter on uses a [semantic-convention](/docs/observe/features/manual-tracing/semantic-conventions) key with a supported value type. +- Anything you filter on uses a [semantic-convention](/docs/sdk/tracing/semantic-conventions) key with a supported value type. ## Causes and fixes | Cause | What you see | Fix | |---|---|---| -| Redaction is on (check first) | Input/output render as hidden or blank, but the span is otherwise complete | Confirm whether `FI_HIDE_INPUTS` / `FI_HIDE_OUTPUTS` or `TraceConfig` masking is set — if so, the blank is expected. See [Mask span attributes](/docs/observe/features/manual-tracing/mask-span-attributes). | +| Redaction is on (check first) | Input/output render as hidden or blank, but the span is otherwise complete | Confirm whether `FI_HIDE_INPUTS` / `FI_HIDE_OUTPUTS` or `TraceConfig` masking is set — if so, the blank is expected. See [Mask span attributes](/docs/sdk/tracing/mask-span-attributes). | | Instrumentor not attached for that framework | A framework's child spans (e.g. nested LangGraph nodes) never appear | Install and `instrument()` the instrumentor for the missing framework, attached to the provider. | | Attribute set on the wrong span / after close | A custom attribute you set isn't on the span | Set attributes while the span is active; a value set after the `with` block closes is dropped. | -| Custom key isn't indexed for filtering | The attribute is on the span but you can't filter by it | Use a [semantic-convention](/docs/observe/features/manual-tracing/semantic-conventions) key where one exists — the UI filters on standard keys. | +| Custom key isn't indexed for filtering | The attribute is on the span but you can't filter by it | Use a [semantic-convention](/docs/sdk/tracing/semantic-conventions) key where one exists — the UI filters on standard keys. | | Unsupported value type | The attribute is silently dropped | Attribute values must be string, bool, int, float, or an array of those. | ## Diagnostic commands @@ -95,10 +92,10 @@ If you're still stuck, contact support@futureagi.com with your `project_name`, t ## Next steps - + How attributes get onto spans. - + Why a field might be intentionally hidden. diff --git a/src/pages/docs/observe/troubleshooting/no-traces-appearing.mdx b/src/pages/docs/observe/troubleshooting/no-traces-appearing.mdx index bbf1a4a8..6e5cf971 100644 --- a/src/pages/docs/observe/troubleshooting/no-traces-appearing.mdx +++ b/src/pages/docs/observe/troubleshooting/no-traces-appearing.mdx @@ -25,14 +25,13 @@ geo: canonical: "/docs/observe/troubleshooting/no-traces-appearing" related: - "/docs/observe/quickstart" - - "/docs/observe/features/llm-tracing" - - "/docs/traceai/troubleshooting/spans-not-exported" - - "/docs/observe/features/manual-tracing/set-up-tracing" + - "/docs/observe/guides/explore-dashboard" + - "/docs/sdk/tracing/set-up-tracing" --- ## Symptom -You instrumented your app with traceAI and ran it, but no trace shows up in the [trace explorer](/docs/observe/features/llm-tracing). Typically: +You instrumented your app with traceAI and ran it, but no trace shows up in the [trace explorer](/docs/observe/guides/explore-dashboard). Typically: - A request ran with no error, but no new row appears in the trace list. - A short script (a one-off `python app.py`) never produces a trace. @@ -57,8 +56,6 @@ The most common cause is a short-lived process that exited before its spans flus | Instrumented after the client was created | Some or all spans never emit because the client wasn't wrapped | Call `register()` and the instrumentor **before** constructing the framework client. | | Date-picker window too narrow | The trace exists but is filtered out of the view | Widen the date range to **Today** and enable **Auto refresh**. | -The Observe trace explorer date-range picker, widened so a recent trace falls inside the selected window - ## Diagnostic commands Confirm the keys are present in the environment the app actually runs in: @@ -83,8 +80,6 @@ trace_provider = register( trace_provider.force_flush() ``` -If spans still never leave the SDK, work through [Spans not exported](/docs/traceai/troubleshooting/spans-not-exported). - ## Minimal smoke test Send one request, then open **Observe → your project → Tracing** with **Auto refresh** on and the date range widened to **Today**. A new trace should appear within seconds, **Status OK**, with input, output, latency, and model populated. If it doesn't, recheck the causes above in order. @@ -92,24 +87,21 @@ Send one request, then open **Observe → your project → Tracing** with **Auto ## Prevent recurrence - Add `trace_provider.force_flush()` to short scripts and job runners. -- Call `register()` + `instrument()` once at startup, before any client is built — see [Set up tracing](/docs/observe/features/manual-tracing/set-up-tracing). +- Call `register()` + `instrument()` once at startup, before any client is built — see [Set up tracing](/docs/sdk/tracing/set-up-tracing). - Keep `FI_API_KEY`, `FI_SECRET_KEY`, and `project_type` in your startup config so they can't drift per environment. If you're still stuck, collect your `project_name`, a request timestamp, your installed `fi-instrumentation-otel` and instrumentor versions, and any stderr, and contact support@futureagi.com. ## Next steps - + Get a first trace flowing end to end. - + The setup this page diagnoses. - - When spans never leave the SDK at all. - - + Where traces should appear. diff --git a/src/pages/docs/prompt/features/linked-traces.mdx b/src/pages/docs/prompt/features/linked-traces.mdx index ed0869a1..10e84ba6 100644 --- a/src/pages/docs/prompt/features/linked-traces.mdx +++ b/src/pages/docs/prompt/features/linked-traces.mdx @@ -40,7 +40,7 @@ Once linked, the Prompt Workbench shows aggregated metrics per prompt version al ## How to -To link prompts to traces, you need to associate the prompt used in a generation with the corresponding trace. The process is described in the observability and manual tracing docs: [Log prompt templates](/docs/observe/features/manual-tracing/log-prompt-templates). Once your application sends traces that include the prompt template (or template ID), Future AGI links those traces to the prompt in the Prompt Workbench. +To link prompts to traces, you need to associate the prompt used in a generation with the corresponding trace. The process is described in the observability and manual tracing docs: [Log prompt templates](/docs/sdk/tracing/log-prompt-templates). Once your application sends traces that include the prompt template (or template ID), Future AGI links those traces to the prompt in the Prompt Workbench. --- @@ -70,7 +70,7 @@ Compare the same metric across **prompt versions** or **time ranges** to see if Manage and fetch prompts programmatically. - + Set up the trace-to-prompt connection in your application. diff --git a/src/pages/docs/quickstart/setup-mcp-server.mdx b/src/pages/docs/quickstart/setup-mcp-server.mdx index a4d08ace..fed183e7 100644 --- a/src/pages/docs/quickstart/setup-mcp-server.mdx +++ b/src/pages/docs/quickstart/setup-mcp-server.mdx @@ -44,7 +44,7 @@ https://api.futureagi.com/mcp With **Future AGI's MCP Server**, you can use natural language to: - **Run automatic evaluations**: evaluate batch and single inputs on various [evaluation](/docs/cookbook/quickstart/first-eval) metrics, both on local datapoints and large datasets -- **Prototype and observe your agents**: add [observability](/docs/observe/features/quickstart), evaluations while [prototyping](/docs/prototype) and deploying agents into production +- **Prototype and observe your agents**: add [observability](/docs/observe/quickstart), evaluations while [prototyping](/docs/prototype) and deploying agents into production - **Manage datasets**: upload, evaluate, download [datasets](/docs/dataset) and find insights - **Add protection rules**: apply toxicity detection, prompt injection protection, and other guardrails automatically - **Generate synthetic data**: describe your dataset and objective to generate synthetic data @@ -54,5 +54,5 @@ Check out our [blog post](https://futureagi.com/blogs/model-context-protocol-mcp ## Next Steps - [Run your first evaluation](/docs/cookbook/quickstart/first-eval) using natural language through the MCP server -- [Explore the Observe quickstart](/docs/observe/features/quickstart) to add tracing to your project +- [Explore the Observe quickstart](/docs/observe/quickstart) to add tracing to your project - [Learn about Protect](/docs/protect) to set up real-time guardrails for your AI application diff --git a/src/pages/docs/quickstart/setup-observability.mdx b/src/pages/docs/quickstart/setup-observability.mdx index 283a71a7..f3b9791f 100644 --- a/src/pages/docs/quickstart/setup-observability.mdx +++ b/src/pages/docs/quickstart/setup-observability.mdx @@ -77,7 +77,7 @@ Observe supports auto-instrumentation for OpenAI, Anthropic, LangChain, LlamaInd There are 2 ways to implement tracing in your project: 1. **Auto Instrumentor**: Automatically captures all LLM calls. Recommended for most use cases. - 2. **Manual Tracing**: Gives you full control over what gets traced using OpenTelemetry. [Learn more](/docs/observe/features/manual-tracing/set-up-tracing) + 2. **Manual Tracing**: Gives you full control over what gets traced using OpenTelemetry. [Learn more](/docs/sdk/tracing/set-up-tracing) Here's a complete example using auto-instrumentation with OpenAI: @@ -136,5 +136,5 @@ Observe supports auto-instrumentation for OpenAI, Anthropic, LangChain, LlamaInd ## Next Steps - [Add more integrations](/docs/integrations) for Anthropic, LangChain, LlamaIndex, and others -- [Set up manual tracing](/docs/observe/features/manual-tracing/set-up-tracing) for custom spans and attributes -- [Add inline evaluations](/docs/observe/features/manual-tracing/in-line-evals) to evaluate traces as they come in \ No newline at end of file +- [Set up manual tracing](/docs/sdk/tracing/set-up-tracing) for custom spans and attributes +- [Add inline evaluations](/docs/sdk/tracing/in-line-evals) to evaluate traces as they come in \ No newline at end of file diff --git a/src/pages/docs/sdk/annotation-queues.mdx b/src/pages/docs/sdk/annotation-queues.mdx deleted file mode 100644 index 5840a517..00000000 --- a/src/pages/docs/sdk/annotation-queues.mdx +++ /dev/null @@ -1,1030 +0,0 @@ ---- -title: "Annotation Queues: Future AGI Python SDK Reference" -description: "Reference for the AnnotationQueue class in the Future AGI Python SDK, covering how to create, fetch, and populate annotation queues programmatically." ---- - -For step-by-step examples, see the [Annotation Queue Using SDK](/docs/annotations/sdk/annotation-queue-using-sdk) guide. - -# `AnnotationQueue` Class - -The `AnnotationQueue` class is the SDK client for managing annotation queues, items, scores, and analytics. Annotation queues let you organize traces, sessions, datasets, and simulation outputs for structured human review. You can define custom labels, set how many annotations are needed per item, and add guidelines to keep feedback consistent. - - -All methods that accept `queue_id` also accept `queue_name` as an alternative. Similarly, methods that accept `label_id` also accept `label_name`. The SDK resolves names to IDs automatically. If multiple matches are found, an error is raised asking you to use the ID instead. - - -## Installation - -```bash -pip install futureagi -``` - -## Initialization - -```python -from fi.queues import AnnotationQueue - -client = AnnotationQueue( - fi_api_key="your_api_key", - fi_secret_key="your_secret_key", - fi_base_url="https://api.futureagi.com", # optional -) -``` - -**Arguments:** - -- `fi_api_key` (Optional[str]): API key for authentication. -- `fi_secret_key` (Optional[str]): Secret key for authentication. -- `fi_base_url` (Optional[str]): Base URL for the API. - ---- - -## Labels - -### `create_label` - -Creates an annotation label. Labels define what annotators evaluate (e.g. sentiment, quality, relevance). - -```python -def create_label( - self, - name: str, - type: str, - *, - settings: Optional[Dict[str, Any]] = None, - description: Optional[str] = None, - project: Optional[str] = None, - timeout: Optional[int] = None, -) -> AnnotationLabel -``` - -- **Arguments:** - - `name` (str): Label name. Must be unique per organization, type, and project. - - `type` (str): Label type — `"categorical"`, `"text"`, `"numeric"`, `"star"`, or `"thumbs_up_down"`. - - `settings` (Optional[Dict[str, Any]]): Type-specific configuration. See [Label Settings by Type](#label-settings-by-type) below. - - `description` (Optional[str]): Description of the label. - - `project` (Optional[str]): Project ID to scope the label to. If omitted, the label is organization-wide. - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `AnnotationLabel` instance - -#### Label Settings by Type - - - - ```python - { - "rule_prompt": "Classify the sentiment", # str, required - "multi_choice": False, # bool, required - "options": [ # list, required (min 2) - {"label": "Positive"}, - {"label": "Negative"}, - {"label": "Neutral"}, - ], - "auto_annotate": False, # bool, required - "strategy": None, # "Rag" or None, required - } - ``` - - - ```python - { - "placeholder": "Enter your feedback...", # str, required - "max_length": 500, # int, required - "min_length": 1, # int, required - } - ``` - - - ```python - { - "min": 0, # number, required - "max": 10, # number, required - "step_size": 1, # number, required - "display_type": "slider", # "slider" or "button", required - } - ``` - - - ```python - { - "no_of_stars": 5, # int, required (>= 1) - } - ``` - - - ```python - {} # No settings required - ``` - - - ---- - -### `list_labels` - -Lists annotation labels available to the organization. - -```python -def list_labels( - self, - *, - project_id: Optional[str] = None, - timeout: Optional[int] = None, -) -> List[AnnotationLabel] -``` - -- **Arguments:** - - `project_id` (Optional[str]): Filter labels by project ID. - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `List[AnnotationLabel]` - ---- - -### `get_label` - -Gets a single annotation label by ID or name. - -```python -def get_label( - self, - label_id: Optional[str] = None, - *, - label_name: Optional[str] = None, - timeout: Optional[int] = None, -) -> AnnotationLabel -``` - -- **Arguments:** - - `label_id` (Optional[str]): UUID of the annotation label. - - `label_name` (Optional[str]): Name of the annotation label (alternative to `label_id`). - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `AnnotationLabel` instance - ---- - -### `delete_label` - -Deletes an annotation label. - -```python -def delete_label( - self, - label_id: Optional[str] = None, - *, - label_name: Optional[str] = None, - timeout: Optional[int] = None, -) -> Dict[str, Any] -``` - -- **Arguments:** - - `label_id` (Optional[str]): UUID of the annotation label. - - `label_name` (Optional[str]): Name of the annotation label (alternative to `label_id`). - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `Dict[str, Any]` - ---- - -### `add_label` - -Attaches an existing annotation label to the queue. - -```python -def add_label( - self, - queue_id: Optional[str] = None, - label_id: Optional[str] = None, - *, - queue_name: Optional[str] = None, - label_name: Optional[str] = None, - timeout: Optional[int] = None, -) -> Dict[str, Any] -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `label_id` (Optional[str]): UUID of the annotation label. - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `label_name` (Optional[str]): Name of the annotation label (alternative to `label_id`). - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `Dict[str, Any]` - ---- - -### `remove_label` - -Removes an annotation label from the queue. - -```python -def remove_label( - self, - queue_id: Optional[str] = None, - label_id: Optional[str] = None, - *, - queue_name: Optional[str] = None, - label_name: Optional[str] = None, - timeout: Optional[int] = None, -) -> Dict[str, Any] -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `label_id` (Optional[str]): UUID of the annotation label. - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `label_name` (Optional[str]): Name of the annotation label (alternative to `label_id`). - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `Dict[str, Any]` - ---- - -## Queue Management - -### `create` - -Creates a new annotation queue. - -```python -def create( - self, - name: str, - *, - description: Optional[str] = None, - instructions: Optional[str] = None, - assignment_strategy: Optional[str] = None, - annotations_required: Optional[int] = None, - reservation_timeout_minutes: Optional[int] = None, - requires_review: Optional[bool] = None, - project: Optional[str] = None, - dataset: Optional[str] = None, - agent_definition: Optional[str] = None, - timeout: Optional[int] = None, -) -> QueueDetail -``` - -- **Arguments:** - - `name` (str): Name of the annotation queue. - - `description` (Optional[str]): Description of the queue's purpose. - - `instructions` (Optional[str]): Guidelines for annotators. - - `assignment_strategy` (Optional[str]): How items are assigned — `"manual"`, `"round_robin"`, or `"load_balanced"`. - - `annotations_required` (Optional[int]): Number of annotations needed per item. - - `reservation_timeout_minutes` (Optional[int]): Time limit (in minutes) for an annotator to complete an item. - - `requires_review` (Optional[bool]): Whether completed annotations require reviewer approval. - - `project` (Optional[str]): Project ID to scope the queue to. - - `dataset` (Optional[str]): Dataset ID to associate with the queue. - - `agent_definition` (Optional[str]): Agent definition ID to associate with the queue. - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `QueueDetail` instance - ---- - -### `list_queues` - -Lists annotation queues with optional filters. - -```python -def list_queues( - self, - *, - status: Optional[str] = None, - search: Optional[str] = None, - include_counts: bool = True, - page: int = 1, - page_size: int = 20, - timeout: Optional[int] = None, -) -> List[QueueDetail] -``` - -- **Arguments:** - - `status` (Optional[str]): Filter by queue status — `"draft"`, `"active"`, `"paused"`, or `"completed"`. - - `search` (Optional[str]): Search queues by name. - - `include_counts` (bool): Whether to include item/completed counts. Defaults to `True`. - - `page` (int): Page number for pagination. Defaults to `1`. - - `page_size` (int): Number of results per page. Defaults to `20`. - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `List[QueueDetail]` - ---- - -### `get` - -Gets a single annotation queue by ID or name. - -```python -def get( - self, - queue_id: Optional[str] = None, - *, - queue_name: Optional[str] = None, - timeout: Optional[int] = None, -) -> QueueDetail -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `QueueDetail` instance - ---- - -### `update` - -Updates an annotation queue. - -```python -def update( - self, - queue_id: Optional[str] = None, - *, - queue_name: Optional[str] = None, - name: Optional[str] = None, - description: Optional[str] = None, - instructions: Optional[str] = None, - assignment_strategy: Optional[str] = None, - annotations_required: Optional[int] = None, - reservation_timeout_minutes: Optional[int] = None, - requires_review: Optional[bool] = None, - timeout: Optional[int] = None, -) -> QueueDetail -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `name` (Optional[str]): Updated queue name. - - `description` (Optional[str]): Updated description. - - `instructions` (Optional[str]): Updated annotator instructions. - - `assignment_strategy` (Optional[str]): Updated assignment strategy. - - `annotations_required` (Optional[int]): Updated annotations required per item. - - `reservation_timeout_minutes` (Optional[int]): Updated reservation timeout. - - `requires_review` (Optional[bool]): Updated review requirement. - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `QueueDetail` instance - ---- - -### `delete` - -Deletes (soft-deletes) an annotation queue. - -```python -def delete( - self, - queue_id: Optional[str] = None, - *, - queue_name: Optional[str] = None, - timeout: Optional[int] = None, -) -> Dict[str, Any] -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `Dict[str, Any]` - ---- - -## Queue Lifecycle - -### `activate` - -Activates a queue, transitioning it from draft to active status. - -```python -def activate( - self, - queue_id: Optional[str] = None, - *, - queue_name: Optional[str] = None, - timeout: Optional[int] = None, -) -> QueueDetail -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `QueueDetail` instance - ---- - -### `complete_queue` - -Marks a queue as completed. - -```python -def complete_queue( - self, - queue_id: Optional[str] = None, - *, - queue_name: Optional[str] = None, - timeout: Optional[int] = None, -) -> QueueDetail -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `QueueDetail` instance - - -Completing a queue does **not** automatically disable its automation rules. If you have active rules, they may continue adding items to the queue, which will re-activate it. Disable or delete automation rules manually before completing the queue if you want to prevent new items from being added. - - ---- - -## Queue Items - -### `add_items` - -Adds items to the queue for annotation. - -```python -def add_items( - self, - queue_id: Optional[str] = None, - items: Optional[List[Dict[str, str]]] = None, - *, - queue_name: Optional[str] = None, - timeout: Optional[int] = None, -) -> AddItemsResponse -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `items` (List[Dict[str, str]]): List of dicts, each with `source_type` and `source_id`. - - Valid `source_type` values: `"trace"`, `"observation_span"`, `"trace_session"`, `"call_execution"`, `"prototype_run"`, `"dataset_row"`. - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `AddItemsResponse` with `added` and `duplicates` counts. - ---- - -### `list_items` - -Lists items in a queue with optional filters. - -```python -def list_items( - self, - queue_id: Optional[str] = None, - *, - queue_name: Optional[str] = None, - status: Optional[str] = None, - assigned_to: Optional[str] = None, - page: int = 1, - page_size: int = 50, - timeout: Optional[int] = None, -) -> List[QueueItem] -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `status` (Optional[str]): Filter by item status — `"pending"`, `"in_progress"`, or `"completed"`. - - `assigned_to` (Optional[str]): Filter by assigned user ID. - - `page` (int): Page number. Defaults to `1`. - - `page_size` (int): Results per page. Defaults to `50`. - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `List[QueueItem]` - ---- - -### `remove_items` - -Bulk-removes items from the queue. - -```python -def remove_items( - self, - queue_id: Optional[str] = None, - item_ids: Optional[List[str]] = None, - *, - queue_name: Optional[str] = None, - timeout: Optional[int] = None, -) -> Dict[str, Any] -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `item_ids` (List[str]): List of item UUIDs to remove. - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `Dict[str, Any]` - ---- - -### `assign_items` - -Assigns items to an annotator. Pass `user_id=None` to unassign. - -```python -def assign_items( - self, - queue_id: Optional[str] = None, - item_ids: Optional[List[str]] = None, - *, - queue_name: Optional[str] = None, - user_id: Optional[str] = None, - timeout: Optional[int] = None, -) -> Dict[str, Any] -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `item_ids` (List[str]): List of item UUIDs to assign. - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `user_id` (Optional[str]): User UUID to assign to. Pass `None` to unassign. - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `Dict[str, Any]` - ---- - -### `complete_item` - -Marks a queue item as completed. - -```python -def complete_item( - self, - queue_id: Optional[str] = None, - item_id: Optional[str] = None, - *, - queue_name: Optional[str] = None, - timeout: Optional[int] = None, -) -> Dict[str, Any] -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `item_id` (str): UUID of the queue item. - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `Dict[str, Any]` - ---- - -### `skip_item` - -Skips a queue item. - -```python -def skip_item( - self, - queue_id: Optional[str] = None, - item_id: Optional[str] = None, - *, - queue_name: Optional[str] = None, - timeout: Optional[int] = None, -) -> Dict[str, Any] -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `item_id` (str): UUID of the queue item. - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `Dict[str, Any]` - ---- - -## Annotations - -### `submit_annotations` - -Submits annotations for a queue item as the authenticated user. - -```python -def submit_annotations( - self, - queue_id: Optional[str] = None, - item_id: Optional[str] = None, - annotations: Optional[List[Dict[str, Any]]] = None, - *, - queue_name: Optional[str] = None, - notes: Optional[str] = None, - timeout: Optional[int] = None, -) -> Dict[str, Any] -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `item_id` (str): UUID of the queue item. - - `annotations` (List[Dict[str, Any]]): List of dicts, each with `label_id` and `value`. - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `notes` (Optional[str]): Free-text notes. - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `Dict[str, Any]` - ---- - -### `import_annotations` - -Imports annotations for a queue item programmatically. Use this when you want to bulk-import annotations from an external source or automated pipeline. - -```python -def import_annotations( - self, - queue_id: Optional[str] = None, - item_id: Optional[str] = None, - annotations: Optional[List[Dict[str, Any]]] = None, - *, - queue_name: Optional[str] = None, - annotator_id: Optional[str] = None, - timeout: Optional[int] = None, -) -> ImportAnnotationsResponse -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `item_id` (str): UUID of the queue item. - - `annotations` (List[Dict[str, Any]]): List of dicts, each with `label_id` and `value`. Optionally include `score_source` (default: `"imported"`). - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `annotator_id` (Optional[str]): User ID to attribute the annotations to. - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `ImportAnnotationsResponse` with `imported` count. - ---- - -### `get_annotations` - -Gets all annotations for a queue item. - -```python -def get_annotations( - self, - queue_id: Optional[str] = None, - item_id: Optional[str] = None, - *, - queue_name: Optional[str] = None, - timeout: Optional[int] = None, -) -> List[Score] -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `item_id` (str): UUID of the queue item. - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `List[Score]` - ---- - -## Scores - -Scores provide a unified annotation model that can be used independently of queues to annotate any source entity. - -### `create_score` - -Creates a single score with upsert semantics. - -```python -def create_score( - self, - source_type: str, - source_id: str, - label_id: Optional[str] = None, - value: Any = None, - *, - label_name: Optional[str] = None, - score_source: str = "api", - notes: Optional[str] = None, - timeout: Optional[int] = None, -) -> Score -``` - -- **Arguments:** - - `source_type` (str): Source entity type — `"trace"`, `"observation_span"`, `"trace_session"`, `"call_execution"`, `"prototype_run"`, or `"dataset_row"`. - - `source_id` (str): UUID of the source entity. - - `label_id` (Optional[str]): UUID of the annotation label. - - `value` (Any): Annotation value (str, float, bool, or list depending on label type). - - `label_name` (Optional[str]): Name of the annotation label (alternative to `label_id`). - - `score_source` (str): Origin of the score — `"human"`, `"api"`, or `"auto"`. Defaults to `"api"`. - - `notes` (Optional[str]): Free-text notes. - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `Score` instance - ---- - -### `create_scores` - -Creates multiple scores on a single source entity in one request. - -```python -def create_scores( - self, - source_type: str, - source_id: str, - scores: List[Dict[str, Any]], - *, - notes: Optional[str] = None, - timeout: Optional[int] = None, -) -> Dict[str, Any] -``` - -- **Arguments:** - - `source_type` (str): Source entity type. - - `source_id` (str): UUID of the source entity. - - `scores` (List[Dict[str, Any]]): List of dicts, each with `label_id`, `value`, and optionally `score_source`. - - `notes` (Optional[str]): Shared free-text notes. - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `Dict[str, Any]` - ---- - -### `get_scores` - -Gets all scores for a given source entity. - -```python -def get_scores( - self, - source_type: str, - source_id: str, - *, - timeout: Optional[int] = None, -) -> List[Score] -``` - -- **Arguments:** - - `source_type` (str): Source entity type. - - `source_id` (str): UUID of the source entity. - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `List[Score]` - ---- - -## Progress & Analytics - -### `get_progress` - -Gets queue progress metrics. - -```python -def get_progress( - self, - queue_id: Optional[str] = None, - *, - queue_name: Optional[str] = None, - timeout: Optional[int] = None, -) -> QueueProgress -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `QueueProgress` instance with `total`, `pending`, `in_progress`, `completed`, `skipped`, `progress_pct`, and `annotator_stats`. - ---- - -### `get_analytics` - -Gets queue analytics including throughput, annotator performance, and label distribution. - -```python -def get_analytics( - self, - queue_id: Optional[str] = None, - *, - queue_name: Optional[str] = None, - timeout: Optional[int] = None, -) -> QueueAnalytics -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `QueueAnalytics` instance with `throughput`, `annotator_performance`, `label_distribution`, `status_breakdown`, and `total`. - ---- - -### `get_agreement` - -Gets inter-annotator agreement metrics for a queue. - -```python -def get_agreement( - self, - queue_id: Optional[str] = None, - *, - queue_name: Optional[str] = None, - timeout: Optional[int] = None, -) -> QueueAgreement -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `QueueAgreement` instance with `overall_agreement`, `per_label`, and `annotator_pairs`. - ---- - -## Export - -### `export` - -Exports queue annotations in JSON or CSV format. - -```python -def export( - self, - queue_id: Optional[str] = None, - *, - queue_name: Optional[str] = None, - export_format: str = "json", - status: Optional[str] = None, - timeout: Optional[int] = None, -) -> Any -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `export_format` (str): Export format — `"json"` or `"csv"`. Defaults to `"json"`. - - `status` (Optional[str]): Filter by item status (e.g. `"completed"`). - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - For JSON: `List[Dict]`. For CSV: raw text string. - ---- - -### `export_to_dataset` - -Exports annotated queue items to a Future AGI dataset. - -```python -def export_to_dataset( - self, - queue_id: Optional[str] = None, - *, - queue_name: Optional[str] = None, - dataset_name: Optional[str] = None, - dataset_id: Optional[str] = None, - status_filter: Optional[str] = None, - timeout: Optional[int] = None, -) -> ExportToDatasetResponse -``` - -- **Arguments:** - - `queue_id` (Optional[str]): UUID of the annotation queue. - - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). - - `dataset_name` (Optional[str]): Name for a new dataset. Mutually exclusive with `dataset_id`. - - `dataset_id` (Optional[str]): UUID of an existing dataset to append to. Mutually exclusive with `dataset_name`. - - `status_filter` (Optional[str]): Item status to export. Defaults to `"completed"`. - - `timeout` (Optional[int]): Request timeout in seconds. -- **Returns:** - - `ExportToDatasetResponse` with `dataset_id`, `dataset_name`, and `rows_created`. - ---- - -## Data Models - -All data models are importable from the SDK and work with IDE autocomplete: - -```python -from fi.queues import ( - AnnotationLabel, QueueDetail, QueueItem, Score, - QueueProgress, QueueAnalytics, QueueAgreement, - AddItemsResponse, ExportToDatasetResponse, ImportAnnotationsResponse, -) -``` - -### `AnnotationLabel` - -| Field | Type | Description | -|-------|------|-------------| -| `id` | `str` | Label UUID | -| `name` | `str` | Label name | -| `type` | `str` | Label type (`categorical`, `text`, `numeric`, `star`, `thumbs_up_down`) | -| `description` | `Optional[str]` | Label description | -| `settings` | `Optional[Dict[str, Any]]` | Type-specific configuration (see [Label Settings by Type](#label-settings-by-type)) | - -### `QueueDetail` - -| Field | Type | Description | -|-------|------|-------------| -| `id` | `str` | Queue UUID | -| `name` | `str` | Queue name | -| `description` | `Optional[str]` | Queue description | -| `instructions` | `Optional[str]` | Annotator instructions | -| `status` | `Optional[str]` | Queue status (`draft`, `active`, `paused`, `completed`) | -| `assignment_strategy` | `Optional[str]` | Assignment strategy (`manual`, `round_robin`, `load_balanced`) | -| `annotations_required` | `Optional[int]` | Annotations needed per item | -| `reservation_timeout_minutes` | `Optional[int]` | Reservation timeout in minutes | -| `requires_review` | `Optional[bool]` | Whether review is required | -| `created_at` | `Optional[str]` | Creation timestamp | -| `updated_at` | `Optional[str]` | Last update timestamp | -| `item_count` | `Optional[int]` | Total items in queue | -| `completed_count` | `Optional[int]` | Completed items count | - -### `QueueItem` - -| Field | Type | Description | -|-------|------|-------------| -| `id` | `str` | Item UUID | -| `source_type` | `Optional[str]` | Source entity type | -| `source_id` | `Optional[str]` | Source entity UUID | -| `status` | `Optional[str]` | Item status (`pending`, `in_progress`, `completed`) | -| `order` | `Optional[int]` | Item order in queue | -| `assigned_to` | `Optional[str]` | Assigned user ID | -| `created_at` | `Optional[str]` | Creation timestamp | - -### `Score` - -| Field | Type | Description | -|-------|------|-------------| -| `id` | `Optional[str]` | Score UUID | -| `label_id` | `Optional[str]` | Label UUID | -| `label_name` | `Optional[str]` | Label display name | -| `value` | `Optional[Any]` | Annotation value | -| `score_source` | `Optional[str]` | Origin (`human`, `api`, `auto`, `imported`) | -| `notes` | `Optional[str]` | Free-text notes | -| `annotator_id` | `Optional[str]` | Annotator user ID | -| `annotator_name` | `Optional[str]` | Annotator display name | -| `source_type` | `Optional[str]` | Source entity type | -| `source_id` | `Optional[str]` | Source entity UUID | -| `created_at` | `Optional[str]` | Creation timestamp | - -### `QueueProgress` - -| Field | Type | Description | -|-------|------|-------------| -| `total` | `int` | Total items | -| `pending` | `int` | Pending items | -| `in_progress` | `int` | In-progress items | -| `completed` | `int` | Completed items | -| `skipped` | `int` | Skipped items | -| `progress_pct` | `Optional[float]` | Completion percentage | -| `annotator_stats` | `Optional[List[Dict]]` | Per-annotator statistics | - -### `QueueAnalytics` - -| Field | Type | Description | -|-------|------|-------------| -| `throughput` | `Optional[Dict]` | Throughput metrics — contains `daily` (list of `{"date", "count"}` entries for the last 30 days), `total_completed` (int), and `avg_per_day` (float) | -| `annotator_performance` | `Optional[List[Dict]]` | Per-annotator performance — each entry has `user_id`, `name`, `completed`, and `last_active` | -| `label_distribution` | `Optional[Dict]` | Distribution of annotations across labels — keyed by label ID, each with `name`, `type`, and `values` (value-to-count mapping) | -| `status_breakdown` | `Optional[Dict[str, int]]` | Item count by status (e.g. `{"pending": 5, "completed": 10}`) | -| `total` | `Optional[int]` | Total items in the queue | - -### `QueueAgreement` - -| Field | Type | Description | -|-------|------|-------------| -| `overall_agreement` | `Optional[float]` | Overall agreement percentage | -| `per_label` | `Optional[List[Dict]]` | Agreement broken down by label | -| `annotator_pairs` | `Optional[List[Dict]]` | Pairwise annotator agreement | - -### `AddItemsResponse` - -| Field | Type | Description | -|-------|------|-------------| -| `added` | `int` | Number of items added | -| `duplicates` | `int` | Number of duplicate items skipped | -| `errors` | `Optional[List[Dict]]` | Any errors encountered | - -### `ExportToDatasetResponse` - -| Field | Type | Description | -|-------|------|-------------| -| `dataset_id` | `Optional[str]` | Dataset UUID | -| `dataset_name` | `Optional[str]` | Dataset name | -| `rows_created` | `Optional[int]` | Number of rows created | - -### `ImportAnnotationsResponse` - -| Field | Type | Description | -|-------|------|-------------| -| `imported` | `int` | Number of annotations imported | diff --git a/src/pages/docs/sdk/annotation-queues/analytics.mdx b/src/pages/docs/sdk/annotation-queues/analytics.mdx new file mode 100644 index 00000000..1c3515b2 --- /dev/null +++ b/src/pages/docs/sdk/annotation-queues/analytics.mdx @@ -0,0 +1,75 @@ +--- +title: "Progress & analytics" +description: "Queue progress, analytics, and inter-annotator agreement." +--- + + +## `get_progress` + +Gets queue progress metrics. + +```python +def get_progress( + self, + queue_id: Optional[str] = None, + *, + queue_name: Optional[str] = None, + timeout: Optional[int] = None, +) -> QueueProgress +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `QueueProgress` instance with `total`, `pending`, `in_progress`, `completed`, `skipped`, `progress_pct`, and `annotator_stats`. + +--- + +## `get_analytics` + +Gets queue analytics including throughput, annotator performance, and label distribution. + +```python +def get_analytics( + self, + queue_id: Optional[str] = None, + *, + queue_name: Optional[str] = None, + timeout: Optional[int] = None, +) -> QueueAnalytics +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `QueueAnalytics` instance with `throughput`, `annotator_performance`, `label_distribution`, `status_breakdown`, and `total`. + +--- + +## `get_agreement` + +Gets inter-annotator agreement metrics for a queue. + +```python +def get_agreement( + self, + queue_id: Optional[str] = None, + *, + queue_name: Optional[str] = None, + timeout: Optional[int] = None, +) -> QueueAgreement +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `QueueAgreement` instance with `overall_agreement`, `per_label`, and `annotator_pairs`. + +--- + diff --git a/src/pages/docs/sdk/annotation-queues/annotations.mdx b/src/pages/docs/sdk/annotation-queues/annotations.mdx new file mode 100644 index 00000000..e5f87921 --- /dev/null +++ b/src/pages/docs/sdk/annotation-queues/annotations.mdx @@ -0,0 +1,89 @@ +--- +title: "Annotations" +description: "Submit, import, and fetch annotations." +--- + + +## `submit_annotations` + +Submits annotations for a queue item as the authenticated user. + +```python +def submit_annotations( + self, + queue_id: Optional[str] = None, + item_id: Optional[str] = None, + annotations: Optional[List[Dict[str, Any]]] = None, + *, + queue_name: Optional[str] = None, + notes: Optional[str] = None, + timeout: Optional[int] = None, +) -> Dict[str, Any] +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `item_id` (str): UUID of the queue item. + - `annotations` (List[Dict[str, Any]]): List of dicts, each with `label_id` and `value`. + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `notes` (Optional[str]): Free-text notes. + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `Dict[str, Any]` + +--- + +## `import_annotations` + +Imports annotations for a queue item programmatically. Use this when you want to bulk-import annotations from an external source or automated pipeline. + +```python +def import_annotations( + self, + queue_id: Optional[str] = None, + item_id: Optional[str] = None, + annotations: Optional[List[Dict[str, Any]]] = None, + *, + queue_name: Optional[str] = None, + annotator_id: Optional[str] = None, + timeout: Optional[int] = None, +) -> ImportAnnotationsResponse +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `item_id` (str): UUID of the queue item. + - `annotations` (List[Dict[str, Any]]): List of dicts, each with `label_id` and `value`. Optionally include `score_source` (default: `"imported"`). + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `annotator_id` (Optional[str]): User ID to attribute the annotations to. + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `ImportAnnotationsResponse` with `imported` count. + +--- + +## `get_annotations` + +Gets all annotations for a queue item. + +```python +def get_annotations( + self, + queue_id: Optional[str] = None, + item_id: Optional[str] = None, + *, + queue_name: Optional[str] = None, + timeout: Optional[int] = None, +) -> List[Score] +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `item_id` (str): UUID of the queue item. + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `List[Score]` + +--- + diff --git a/src/pages/docs/sdk/annotation-queues/data-models.mdx b/src/pages/docs/sdk/annotation-queues/data-models.mdx new file mode 100644 index 00000000..43a2b913 --- /dev/null +++ b/src/pages/docs/sdk/annotation-queues/data-models.mdx @@ -0,0 +1,123 @@ +--- +title: "Data models" +description: "Response and model types returned by the AnnotationQueue SDK." +--- + + +All data models are importable from the SDK and work with IDE autocomplete: + +```python +from fi.queues import ( + AnnotationLabel, QueueDetail, QueueItem, Score, + QueueProgress, QueueAnalytics, QueueAgreement, + AddItemsResponse, ExportToDatasetResponse, ImportAnnotationsResponse, +) +``` + +## `AnnotationLabel` + +| Field | Type | Description | +|-------|------|-------------| +| `id` | `str` | Label UUID | +| `name` | `str` | Label name | +| `type` | `str` | Label type (`categorical`, `text`, `numeric`, `star`, `thumbs_up_down`) | +| `description` | `Optional[str]` | Label description | +| `settings` | `Optional[Dict[str, Any]]` | Type-specific configuration (see [Label Settings by Type](/docs/sdk/annotation-queues/labels#label-settings-by-type)) | + +## `QueueDetail` + +| Field | Type | Description | +|-------|------|-------------| +| `id` | `str` | Queue UUID | +| `name` | `str` | Queue name | +| `description` | `Optional[str]` | Queue description | +| `instructions` | `Optional[str]` | Annotator instructions | +| `status` | `Optional[str]` | Queue status (`draft`, `active`, `paused`, `completed`) | +| `assignment_strategy` | `Optional[str]` | Assignment strategy (`manual`, `round_robin`, `load_balanced`) | +| `annotations_required` | `Optional[int]` | Annotations needed per item | +| `reservation_timeout_minutes` | `Optional[int]` | Reservation timeout in minutes | +| `requires_review` | `Optional[bool]` | Whether review is required | +| `created_at` | `Optional[str]` | Creation timestamp | +| `updated_at` | `Optional[str]` | Last update timestamp | +| `item_count` | `Optional[int]` | Total items in queue | +| `completed_count` | `Optional[int]` | Completed items count | + +## `QueueItem` + +| Field | Type | Description | +|-------|------|-------------| +| `id` | `str` | Item UUID | +| `source_type` | `Optional[str]` | Source entity type | +| `source_id` | `Optional[str]` | Source entity UUID | +| `status` | `Optional[str]` | Item status (`pending`, `in_progress`, `completed`) | +| `order` | `Optional[int]` | Item order in queue | +| `assigned_to` | `Optional[str]` | Assigned user ID | +| `created_at` | `Optional[str]` | Creation timestamp | + +## `Score` + +| Field | Type | Description | +|-------|------|-------------| +| `id` | `Optional[str]` | Score UUID | +| `label_id` | `Optional[str]` | Label UUID | +| `label_name` | `Optional[str]` | Label display name | +| `value` | `Optional[Any]` | Annotation value | +| `score_source` | `Optional[str]` | Origin (`human`, `api`, `auto`, `imported`) | +| `notes` | `Optional[str]` | Free-text notes | +| `annotator_id` | `Optional[str]` | Annotator user ID | +| `annotator_name` | `Optional[str]` | Annotator display name | +| `source_type` | `Optional[str]` | Source entity type | +| `source_id` | `Optional[str]` | Source entity UUID | +| `created_at` | `Optional[str]` | Creation timestamp | + +## `QueueProgress` + +| Field | Type | Description | +|-------|------|-------------| +| `total` | `int` | Total items | +| `pending` | `int` | Pending items | +| `in_progress` | `int` | In-progress items | +| `completed` | `int` | Completed items | +| `skipped` | `int` | Skipped items | +| `progress_pct` | `Optional[float]` | Completion percentage | +| `annotator_stats` | `Optional[List[Dict]]` | Per-annotator statistics | + +## `QueueAnalytics` + +| Field | Type | Description | +|-------|------|-------------| +| `throughput` | `Optional[Dict]` | Throughput metrics — contains `daily` (list of `{"date", "count"}` entries for the last 30 days), `total_completed` (int), and `avg_per_day` (float) | +| `annotator_performance` | `Optional[List[Dict]]` | Per-annotator performance — each entry has `user_id`, `name`, `completed`, and `last_active` | +| `label_distribution` | `Optional[Dict]` | Distribution of annotations across labels — keyed by label ID, each with `name`, `type`, and `values` (value-to-count mapping) | +| `status_breakdown` | `Optional[Dict[str, int]]` | Item count by status (e.g. `{"pending": 5, "completed": 10}`) | +| `total` | `Optional[int]` | Total items in the queue | + +## `QueueAgreement` + +| Field | Type | Description | +|-------|------|-------------| +| `overall_agreement` | `Optional[float]` | Overall agreement percentage | +| `per_label` | `Optional[List[Dict]]` | Agreement broken down by label | +| `annotator_pairs` | `Optional[List[Dict]]` | Pairwise annotator agreement | + +## `AddItemsResponse` + +| Field | Type | Description | +|-------|------|-------------| +| `added` | `int` | Number of items added | +| `duplicates` | `int` | Number of duplicate items skipped | +| `errors` | `Optional[List[Dict]]` | Any errors encountered | + +## `ExportToDatasetResponse` + +| Field | Type | Description | +|-------|------|-------------| +| `dataset_id` | `Optional[str]` | Dataset UUID | +| `dataset_name` | `Optional[str]` | Dataset name | +| `rows_created` | `Optional[int]` | Number of rows created | + +## `ImportAnnotationsResponse` + +| Field | Type | Description | +|-------|------|-------------| +| `imported` | `int` | Number of annotations imported | diff --git a/src/pages/docs/sdk/annotation-queues/export.mdx b/src/pages/docs/sdk/annotation-queues/export.mdx new file mode 100644 index 00000000..0963972c --- /dev/null +++ b/src/pages/docs/sdk/annotation-queues/export.mdx @@ -0,0 +1,62 @@ +--- +title: "Export" +description: "Export annotations, or export a queue to a dataset." +--- + + +## `export` + +Exports queue annotations in JSON or CSV format. + +```python +def export( + self, + queue_id: Optional[str] = None, + *, + queue_name: Optional[str] = None, + export_format: str = "json", + status: Optional[str] = None, + timeout: Optional[int] = None, +) -> Any +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `export_format` (str): Export format — `"json"` or `"csv"`. Defaults to `"json"`. + - `status` (Optional[str]): Filter by item status (e.g. `"completed"`). + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - For JSON: `List[Dict]`. For CSV: raw text string. + +--- + +## `export_to_dataset` + +Exports annotated queue items to a Future AGI dataset. + +```python +def export_to_dataset( + self, + queue_id: Optional[str] = None, + *, + queue_name: Optional[str] = None, + dataset_name: Optional[str] = None, + dataset_id: Optional[str] = None, + status_filter: Optional[str] = None, + timeout: Optional[int] = None, +) -> ExportToDatasetResponse +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `dataset_name` (Optional[str]): Name for a new dataset. Mutually exclusive with `dataset_id`. + - `dataset_id` (Optional[str]): UUID of an existing dataset to append to. Mutually exclusive with `dataset_name`. + - `status_filter` (Optional[str]): Item status to export. Defaults to `"completed"`. + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `ExportToDatasetResponse` with `dataset_id`, `dataset_name`, and `rows_created`. + +--- + diff --git a/src/pages/docs/sdk/annotation-queues/index.mdx b/src/pages/docs/sdk/annotation-queues/index.mdx new file mode 100644 index 00000000..f2e07bde --- /dev/null +++ b/src/pages/docs/sdk/annotation-queues/index.mdx @@ -0,0 +1,37 @@ +--- +title: "AnnotationQueue class" +description: "Reference for the AnnotationQueue class in the Future AGI Python SDK: create, fetch, and populate annotation queues programmatically." +--- + +For step-by-step examples, see the [Annotation Queue Using SDK](/docs/annotations/sdk/annotation-queue-using-sdk) guide. + + +The `AnnotationQueue` class is the SDK client for managing annotation queues, items, scores, and analytics. Annotation queues let you organize traces, sessions, datasets, and simulation outputs for structured human review. You can define custom labels, set how many annotations are needed per item, and add guidelines to keep feedback consistent. + + +All methods that accept `queue_id` also accept `queue_name` as an alternative. Similarly, methods that accept `label_id` also accept `label_name`. The SDK resolves names to IDs automatically. If multiple matches are found, an error is raised asking you to use the ID instead. + + +## Installation + +```bash +pip install futureagi +``` + +## Initialization + +```python +from fi.queues import AnnotationQueue + +client = AnnotationQueue( + fi_api_key="your_api_key", + fi_secret_key="your_secret_key", + fi_base_url="https://api.futureagi.com", # optional +) +``` + +**Arguments:** + +- `fi_api_key` (Optional[str]): API key for authentication. +- `fi_secret_key` (Optional[str]): Secret key for authentication. +- `fi_base_url` (Optional[str]): Base URL for the API. diff --git a/src/pages/docs/sdk/annotation-queues/items.mdx b/src/pages/docs/sdk/annotation-queues/items.mdx new file mode 100644 index 00000000..66b7c8d5 --- /dev/null +++ b/src/pages/docs/sdk/annotation-queues/items.mdx @@ -0,0 +1,165 @@ +--- +title: "Queue items" +description: "Add, list, remove, assign, complete, and skip queue items." +--- + + +## `add_items` + +Adds items to the queue for annotation. + +```python +def add_items( + self, + queue_id: Optional[str] = None, + items: Optional[List[Dict[str, str]]] = None, + *, + queue_name: Optional[str] = None, + timeout: Optional[int] = None, +) -> AddItemsResponse +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `items` (List[Dict[str, str]]): List of dicts, each with `source_type` and `source_id`. + - Valid `source_type` values: `"trace"`, `"observation_span"`, `"trace_session"`, `"call_execution"`, `"prototype_run"`, `"dataset_row"`. + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `AddItemsResponse` with `added` and `duplicates` counts. + +--- + +## `list_items` + +Lists items in a queue with optional filters. + +```python +def list_items( + self, + queue_id: Optional[str] = None, + *, + queue_name: Optional[str] = None, + status: Optional[str] = None, + assigned_to: Optional[str] = None, + page: int = 1, + page_size: int = 50, + timeout: Optional[int] = None, +) -> List[QueueItem] +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `status` (Optional[str]): Filter by item status — `"pending"`, `"in_progress"`, or `"completed"`. + - `assigned_to` (Optional[str]): Filter by assigned user ID. + - `page` (int): Page number. Defaults to `1`. + - `page_size` (int): Results per page. Defaults to `50`. + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `List[QueueItem]` + +--- + +## `remove_items` + +Bulk-removes items from the queue. + +```python +def remove_items( + self, + queue_id: Optional[str] = None, + item_ids: Optional[List[str]] = None, + *, + queue_name: Optional[str] = None, + timeout: Optional[int] = None, +) -> Dict[str, Any] +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `item_ids` (List[str]): List of item UUIDs to remove. + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `Dict[str, Any]` + +--- + +## `assign_items` + +Assigns items to an annotator. Pass `user_id=None` to unassign. + +```python +def assign_items( + self, + queue_id: Optional[str] = None, + item_ids: Optional[List[str]] = None, + *, + queue_name: Optional[str] = None, + user_id: Optional[str] = None, + timeout: Optional[int] = None, +) -> Dict[str, Any] +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `item_ids` (List[str]): List of item UUIDs to assign. + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `user_id` (Optional[str]): User UUID to assign to. Pass `None` to unassign. + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `Dict[str, Any]` + +--- + +## `complete_item` + +Marks a queue item as completed. + +```python +def complete_item( + self, + queue_id: Optional[str] = None, + item_id: Optional[str] = None, + *, + queue_name: Optional[str] = None, + timeout: Optional[int] = None, +) -> Dict[str, Any] +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `item_id` (str): UUID of the queue item. + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `Dict[str, Any]` + +--- + +## `skip_item` + +Skips a queue item. + +```python +def skip_item( + self, + queue_id: Optional[str] = None, + item_id: Optional[str] = None, + *, + queue_name: Optional[str] = None, + timeout: Optional[int] = None, +) -> Dict[str, Any] +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `item_id` (str): UUID of the queue item. + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `Dict[str, Any]` + +--- + diff --git a/src/pages/docs/sdk/annotation-queues/labels.mdx b/src/pages/docs/sdk/annotation-queues/labels.mdx new file mode 100644 index 00000000..7883b6f3 --- /dev/null +++ b/src/pages/docs/sdk/annotation-queues/labels.mdx @@ -0,0 +1,207 @@ +--- +title: "Labels" +description: "Create, list, get, delete, add, and remove annotation labels." +--- + + +## `create_label` + +Creates an annotation label. Labels define what annotators evaluate (e.g. sentiment, quality, relevance). + +```python +def create_label( + self, + name: str, + type: str, + *, + settings: Optional[Dict[str, Any]] = None, + description: Optional[str] = None, + project: Optional[str] = None, + timeout: Optional[int] = None, +) -> AnnotationLabel +``` + +- **Arguments:** + - `name` (str): Label name. Must be unique per organization, type, and project. + - `type` (str): Label type — `"categorical"`, `"text"`, `"numeric"`, `"star"`, or `"thumbs_up_down"`. + - `settings` (Optional[Dict[str, Any]]): Type-specific configuration. See [Label Settings by Type](#label-settings-by-type) below. + - `description` (Optional[str]): Description of the label. + - `project` (Optional[str]): Project ID to scope the label to. If omitted, the label is organization-wide. + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `AnnotationLabel` instance + +#### Label Settings by Type + + + + ```python + { + "rule_prompt": "Classify the sentiment", # str, required + "multi_choice": False, # bool, required + "options": [ # list, required (min 2) + {"label": "Positive"}, + {"label": "Negative"}, + {"label": "Neutral"}, + ], + "auto_annotate": False, # bool, required + "strategy": None, # "Rag" or None, required + } + ``` + + + ```python + { + "placeholder": "Enter your feedback...", # str, required + "max_length": 500, # int, required + "min_length": 1, # int, required + } + ``` + + + ```python + { + "min": 0, # number, required + "max": 10, # number, required + "step_size": 1, # number, required + "display_type": "slider", # "slider" or "button", required + } + ``` + + + ```python + { + "no_of_stars": 5, # int, required (>= 1) + } + ``` + + + ```python + {} # No settings required + ``` + + + +--- + +## `list_labels` + +Lists annotation labels available to the organization. + +```python +def list_labels( + self, + *, + project_id: Optional[str] = None, + timeout: Optional[int] = None, +) -> List[AnnotationLabel] +``` + +- **Arguments:** + - `project_id` (Optional[str]): Filter labels by project ID. + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `List[AnnotationLabel]` + +--- + +## `get_label` + +Gets a single annotation label by ID or name. + +```python +def get_label( + self, + label_id: Optional[str] = None, + *, + label_name: Optional[str] = None, + timeout: Optional[int] = None, +) -> AnnotationLabel +``` + +- **Arguments:** + - `label_id` (Optional[str]): UUID of the annotation label. + - `label_name` (Optional[str]): Name of the annotation label (alternative to `label_id`). + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `AnnotationLabel` instance + +--- + +## `delete_label` + +Deletes an annotation label. + +```python +def delete_label( + self, + label_id: Optional[str] = None, + *, + label_name: Optional[str] = None, + timeout: Optional[int] = None, +) -> Dict[str, Any] +``` + +- **Arguments:** + - `label_id` (Optional[str]): UUID of the annotation label. + - `label_name` (Optional[str]): Name of the annotation label (alternative to `label_id`). + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `Dict[str, Any]` + +--- + +## `add_label` + +Attaches an existing annotation label to the queue. + +```python +def add_label( + self, + queue_id: Optional[str] = None, + label_id: Optional[str] = None, + *, + queue_name: Optional[str] = None, + label_name: Optional[str] = None, + timeout: Optional[int] = None, +) -> Dict[str, Any] +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `label_id` (Optional[str]): UUID of the annotation label. + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `label_name` (Optional[str]): Name of the annotation label (alternative to `label_id`). + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `Dict[str, Any]` + +--- + +## `remove_label` + +Removes an annotation label from the queue. + +```python +def remove_label( + self, + queue_id: Optional[str] = None, + label_id: Optional[str] = None, + *, + queue_name: Optional[str] = None, + label_name: Optional[str] = None, + timeout: Optional[int] = None, +) -> Dict[str, Any] +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `label_id` (Optional[str]): UUID of the annotation label. + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `label_name` (Optional[str]): Name of the annotation label (alternative to `label_id`). + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `Dict[str, Any]` + +--- + diff --git a/src/pages/docs/sdk/annotation-queues/lifecycle.mdx b/src/pages/docs/sdk/annotation-queues/lifecycle.mdx new file mode 100644 index 00000000..cc378a07 --- /dev/null +++ b/src/pages/docs/sdk/annotation-queues/lifecycle.mdx @@ -0,0 +1,56 @@ +--- +title: "Queue lifecycle" +description: "Activate and complete annotation queues." +--- + + +## `activate` + +Activates a queue, transitioning it from draft to active status. + +```python +def activate( + self, + queue_id: Optional[str] = None, + *, + queue_name: Optional[str] = None, + timeout: Optional[int] = None, +) -> QueueDetail +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `QueueDetail` instance + +--- + +## `complete_queue` + +Marks a queue as completed. + +```python +def complete_queue( + self, + queue_id: Optional[str] = None, + *, + queue_name: Optional[str] = None, + timeout: Optional[int] = None, +) -> QueueDetail +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `QueueDetail` instance + + +Completing a queue does **not** automatically disable its automation rules. If you have active rules, they may continue adding items to the queue, which will re-activate it. Disable or delete automation rules manually before completing the queue if you want to prevent new items from being added. + + +--- + diff --git a/src/pages/docs/sdk/annotation-queues/queues.mdx b/src/pages/docs/sdk/annotation-queues/queues.mdx new file mode 100644 index 00000000..71b5a0e1 --- /dev/null +++ b/src/pages/docs/sdk/annotation-queues/queues.mdx @@ -0,0 +1,157 @@ +--- +title: "Queue management" +description: "Create, list, get, update, and delete annotation queues." +--- + + +## `create` + +Creates a new annotation queue. + +```python +def create( + self, + name: str, + *, + description: Optional[str] = None, + instructions: Optional[str] = None, + assignment_strategy: Optional[str] = None, + annotations_required: Optional[int] = None, + reservation_timeout_minutes: Optional[int] = None, + requires_review: Optional[bool] = None, + project: Optional[str] = None, + dataset: Optional[str] = None, + agent_definition: Optional[str] = None, + timeout: Optional[int] = None, +) -> QueueDetail +``` + +- **Arguments:** + - `name` (str): Name of the annotation queue. + - `description` (Optional[str]): Description of the queue's purpose. + - `instructions` (Optional[str]): Guidelines for annotators. + - `assignment_strategy` (Optional[str]): How items are assigned — `"manual"`, `"round_robin"`, or `"load_balanced"`. + - `annotations_required` (Optional[int]): Number of annotations needed per item. + - `reservation_timeout_minutes` (Optional[int]): Time limit (in minutes) for an annotator to complete an item. + - `requires_review` (Optional[bool]): Whether completed annotations require reviewer approval. + - `project` (Optional[str]): Project ID to scope the queue to. + - `dataset` (Optional[str]): Dataset ID to associate with the queue. + - `agent_definition` (Optional[str]): Agent definition ID to associate with the queue. + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `QueueDetail` instance + +--- + +## `list_queues` + +Lists annotation queues with optional filters. + +```python +def list_queues( + self, + *, + status: Optional[str] = None, + search: Optional[str] = None, + include_counts: bool = True, + page: int = 1, + page_size: int = 20, + timeout: Optional[int] = None, +) -> List[QueueDetail] +``` + +- **Arguments:** + - `status` (Optional[str]): Filter by queue status — `"draft"`, `"active"`, `"paused"`, or `"completed"`. + - `search` (Optional[str]): Search queues by name. + - `include_counts` (bool): Whether to include item/completed counts. Defaults to `True`. + - `page` (int): Page number for pagination. Defaults to `1`. + - `page_size` (int): Number of results per page. Defaults to `20`. + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `List[QueueDetail]` + +--- + +## `get` + +Gets a single annotation queue by ID or name. + +```python +def get( + self, + queue_id: Optional[str] = None, + *, + queue_name: Optional[str] = None, + timeout: Optional[int] = None, +) -> QueueDetail +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `QueueDetail` instance + +--- + +## `update` + +Updates an annotation queue. + +```python +def update( + self, + queue_id: Optional[str] = None, + *, + queue_name: Optional[str] = None, + name: Optional[str] = None, + description: Optional[str] = None, + instructions: Optional[str] = None, + assignment_strategy: Optional[str] = None, + annotations_required: Optional[int] = None, + reservation_timeout_minutes: Optional[int] = None, + requires_review: Optional[bool] = None, + timeout: Optional[int] = None, +) -> QueueDetail +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `name` (Optional[str]): Updated queue name. + - `description` (Optional[str]): Updated description. + - `instructions` (Optional[str]): Updated annotator instructions. + - `assignment_strategy` (Optional[str]): Updated assignment strategy. + - `annotations_required` (Optional[int]): Updated annotations required per item. + - `reservation_timeout_minutes` (Optional[int]): Updated reservation timeout. + - `requires_review` (Optional[bool]): Updated review requirement. + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `QueueDetail` instance + +--- + +## `delete` + +Deletes (soft-deletes) an annotation queue. + +```python +def delete( + self, + queue_id: Optional[str] = None, + *, + queue_name: Optional[str] = None, + timeout: Optional[int] = None, +) -> Dict[str, Any] +``` + +- **Arguments:** + - `queue_id` (Optional[str]): UUID of the annotation queue. + - `queue_name` (Optional[str]): Name of the annotation queue (alternative to `queue_id`). + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `Dict[str, Any]` + +--- + diff --git a/src/pages/docs/sdk/annotation-queues/scores.mdx b/src/pages/docs/sdk/annotation-queues/scores.mdx new file mode 100644 index 00000000..00908bd4 --- /dev/null +++ b/src/pages/docs/sdk/annotation-queues/scores.mdx @@ -0,0 +1,91 @@ +--- +title: "Scores" +description: "Create and fetch scores for queue items." +--- + + +Scores provide a unified annotation model that can be used independently of queues to annotate any source entity. + +## `create_score` + +Creates a single score with upsert semantics. + +```python +def create_score( + self, + source_type: str, + source_id: str, + label_id: Optional[str] = None, + value: Any = None, + *, + label_name: Optional[str] = None, + score_source: str = "api", + notes: Optional[str] = None, + timeout: Optional[int] = None, +) -> Score +``` + +- **Arguments:** + - `source_type` (str): Source entity type — `"trace"`, `"observation_span"`, `"trace_session"`, `"call_execution"`, `"prototype_run"`, or `"dataset_row"`. + - `source_id` (str): UUID of the source entity. + - `label_id` (Optional[str]): UUID of the annotation label. + - `value` (Any): Annotation value (str, float, bool, or list depending on label type). + - `label_name` (Optional[str]): Name of the annotation label (alternative to `label_id`). + - `score_source` (str): Origin of the score — `"human"`, `"api"`, or `"auto"`. Defaults to `"api"`. + - `notes` (Optional[str]): Free-text notes. + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `Score` instance + +--- + +## `create_scores` + +Creates multiple scores on a single source entity in one request. + +```python +def create_scores( + self, + source_type: str, + source_id: str, + scores: List[Dict[str, Any]], + *, + notes: Optional[str] = None, + timeout: Optional[int] = None, +) -> Dict[str, Any] +``` + +- **Arguments:** + - `source_type` (str): Source entity type. + - `source_id` (str): UUID of the source entity. + - `scores` (List[Dict[str, Any]]): List of dicts, each with `label_id`, `value`, and optionally `score_source`. + - `notes` (Optional[str]): Shared free-text notes. + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `Dict[str, Any]` + +--- + +## `get_scores` + +Gets all scores for a given source entity. + +```python +def get_scores( + self, + source_type: str, + source_id: str, + *, + timeout: Optional[int] = None, +) -> List[Score] +``` + +- **Arguments:** + - `source_type` (str): Source entity type. + - `source_id` (str): UUID of the source entity. + - `timeout` (Optional[int]): Request timeout in seconds. +- **Returns:** + - `List[Score]` + +--- + diff --git a/src/pages/docs/sdk/index.mdx b/src/pages/docs/sdk/index.mdx index 503bed0a..2e003aa7 100644 --- a/src/pages/docs/sdk/index.mdx +++ b/src/pages/docs/sdk/index.mdx @@ -176,172 +176,18 @@ fi-instrumentation-otel ← standalone tracing layer Tracing only. Java has 25+ instrumentors (Maven via JitPack, group ID `com.github.future-agi.traceAI`). C# has a single NuGet package (`fi-instrumentation-otel`). See the [Tracing reference](/docs/sdk/tracing) for details. -## Evaluations — `ai-evaluation` +## List of SDKs -76+ local metrics for things like tone, hallucination, bias, and factual accuracy. Also includes guardrails (toxicity, PII, prompt injection) that run in under 10ms. +Each SDK installs independently. Pick the ones you need and follow its reference for the full API. -Available in Python and TypeScript. - - - - All 76+ local metrics — browse by category, see config options, and run examples. - - - Real-time guardrails for toxicity, PII, prompt injection, and content moderation. - - - - - - | Extra | Install | What it adds | - |-------|---------|-------------| - | NLI models | `pip install ai-evaluation[nli]` | DeBERTa for faithfulness and hallucination detection | - | Embeddings | `pip install ai-evaluation[embeddings]` | Sentence-transformers for semantic similarity | - | Feedback | `pip install ai-evaluation[feedback]` | ChromaDB-backed feedback collection | - | Distributed | `pip install ai-evaluation[celery]` | Celery + Redis for distributed eval runs | - | Everything | `pip install ai-evaluation[all]` | All optional dependencies | - - - -## Tracing — `fi-instrumentation-otel` + `traceai-*` - -Install the core library plus one instrumentor per framework. LLM calls, retrieval steps, and agent actions get traced and sent to your Future AGI dashboard. - -Available in Python, TypeScript, Java, and C#. - -```python -from fi_instrumentation import register -from fi_instrumentation.fi_types import ProjectType - -trace_provider = register( - project_name="my-project", - project_type=ProjectType.OBSERVE, -) - -from traceai_openai import OpenAIInstrumentor -OpenAIInstrumentor().instrument(tracer_provider=trace_provider) - -# All OpenAI calls are now traced -# Traces appear in your Future AGI dashboard under "my-project" -``` - - - - | Package | Framework | - |---------|-----------| - | `traceai-openai` | OpenAI | - | `traceai-anthropic` | Anthropic | - | `traceai-google-genai` | Google Generative AI | - | `traceai-vertexai` | Google Vertex AI | - | `traceai-bedrock` | AWS Bedrock | - | `traceai-mistralai` | Mistral AI | - | `traceai-groq` | Groq | - | `traceai-litellm` | LiteLLM | - | `traceai-cohere` | Cohere | - | `traceai-ollama` | Ollama | - | `traceai-deepseek` | DeepSeek | - | `traceai-together` | Together AI | - | `traceai-fireworks` | Fireworks AI | - | `traceai-cerebras` | Cerebras | - | `traceai-xai` | xAI / Grok | - | `traceai-vllm` | vLLM | - | `traceai-portkey` | Portkey | - | `traceai-huggingface` | HuggingFace | - - - | Package | Framework | - |---------|-----------| - | `traceai-langchain` | LangChain / LangGraph | - | `traceai-llamaindex` | LlamaIndex | - | `traceai-crewai` | CrewAI | - | `traceai-openai-agents` | OpenAI Agents SDK | - | `traceai-autogen` | Microsoft AutoGen | - | `traceai-smolagents` | HuggingFace SmolAgents | - | `traceai-google-adk` | Google Agent Dev Kit | - | `traceai-claude-agent-sdk` | Claude Agent SDK | - | `traceai-pydantic-ai` | Pydantic AI | - | `traceai-strands` | AWS Strands Agents | - | `traceai-agno` | Agno | - | `traceai-beeai` | IBM BeeAI | - | `traceai-haystack` | Haystack | - | `traceai-dspy` | DSPy | - | `traceai-guardrails` | Guardrails AI | - | `traceai-instructor` | Instructor | - | `traceai-mcp` | Model Context Protocol | - - - | Package | Framework | - |---------|-----------| - | `traceai-pipecat` | Pipecat | - | `traceai-livekit` | LiveKit | - - - | Package | Framework | - |---------|-----------| - | `traceai-pinecone` | Pinecone | - | `traceai-chromadb` | ChromaDB | - | `traceai-qdrant` | Qdrant | - | `traceai-weaviate` | Weaviate | - | `traceai-milvus` | Milvus | - | `traceai-lancedb` | LanceDB | - | `traceai-mongodb` | MongoDB | - | `traceai-pgvector` | pgvector | - | `traceai-redis` | Redis | - - - - - Each instrumentor is lightweight and independent. Only install the ones for frameworks you actually use. - - -## Core SDK — `futureagi` - -Datasets, prompt versioning, and knowledge bases. If you installed `ai-evaluation`, you already have this. - -Available in Python and TypeScript. - - - - Create, version, and manage test datasets. Import from CSV, DataFrames, or HuggingFace. - - - Upload documents to build knowledge bases for RAG evaluation and context injection. - - - -## Prompt Optimization — `agent-opt` - -Six optimization algorithms: Random Search, Bayesian, ProTeGi, Meta-Prompt, PromptWizard, and GEPA. Each uses eval metrics to score prompt variants and find the best one. - -Python only. - -```bash -pip install agent-opt -``` - -## Simulation Testing — `agent-simulate` - -Run simulated conversations against your voice AI agents using configurable personas. Captures audio, transcripts, and eval scores. - -Python only. - -```bash -pip install agent-simulate -``` - -## Next Steps - - - - Explore 76+ local metrics by category. - - - Set up observability for your AI stack. + + + 76+ local metrics for tone, hallucination, bias, and factual accuracy, plus guardrails that run in under 10ms. - - Create and manage datasets for evaluation. + + Auto-instrument 45+ frameworks or add custom spans. LLM calls, retrieval, and agent actions stream to your dashboard. - - Real-time guardrails for toxicity, PII, and prompt injection. + + Datasets, Prompt Optimization, Simulation, Knowledge Base, and Protect. Install and quick start for each. diff --git a/src/pages/docs/sdk/list/core.mdx b/src/pages/docs/sdk/list/core.mdx new file mode 100644 index 00000000..7179756d --- /dev/null +++ b/src/pages/docs/sdk/list/core.mdx @@ -0,0 +1,56 @@ +--- +title: "Core SDKs" +description: "Datasets, Prompt Optimization, Simulation, Knowledge Base, and Protect. Install and quick start for each, with links to the full reference." +--- + +Install and quick start for the rest of the Future AGI SDKs. Each links to its full reference for the complete API. + +## Datasets + +Create, version, and manage test datasets and prompts with `futureagi`. Import from CSV, DataFrames, or HuggingFace. If you installed `ai-evaluation`, you already have it. Available in Python and TypeScript. + +```bash +pip install futureagi +``` + +**Full reference:** [Datasets SDK](/docs/sdk/datasets) + +## Prompt Optimization + +Optimize prompts automatically with `agent-opt`: six algorithms (Random Search, Bayesian, ProTeGi, Meta-Prompt, PromptWizard, and GEPA) that use eval metrics to score prompt variants and pick the best one. Python only. + +```bash +pip install agent-opt +``` + +**Full reference:** [Prompt Optimization SDK](/docs/sdk/optimization) + +## Simulation Testing + +Run simulated conversations against your voice AI agents with `agent-simulate`, using configurable personas. Captures audio, transcripts, and eval scores. Python only. + +```bash +pip install agent-simulate +``` + +**Full reference:** [Simulation SDK](/docs/sdk/simulate) + +## Knowledge Base + +Upload documents to build knowledge bases for RAG evaluation and context injection with `futureagi`. If you installed `ai-evaluation`, you already have it. Available in Python and TypeScript. + +```bash +pip install futureagi +``` + +**Full reference:** [Knowledge Base SDK](/docs/sdk/knowledgebase) + +## Protect + +Guard LLM inputs and outputs with the Protect module in `ai-evaluation`: real-time checks for toxicity, PII, prompt injection, and content moderation that run in under 10ms. Available in Python. + +```bash +pip install ai-evaluation +``` + +**Full reference:** [Protect SDK](/docs/sdk/protect) diff --git a/src/pages/docs/sdk/list/evaluation.mdx b/src/pages/docs/sdk/list/evaluation.mdx new file mode 100644 index 00000000..3248913b --- /dev/null +++ b/src/pages/docs/sdk/list/evaluation.mdx @@ -0,0 +1,45 @@ +--- +title: "Evaluations SDK" +description: "Evaluate LLM outputs with 76+ local metrics, cloud Turing models, and guardrails in the ai-evaluation package." +--- + +Evaluate LLM outputs with the `ai-evaluation` package: 76+ local metrics for tone, hallucination, bias, and factual accuracy, plus guardrails (toxicity, PII, prompt injection) that run in under 10ms. Available in Python and TypeScript. + +```bash +pip install ai-evaluation +``` + +```python +from fi.evals import evaluate + +# Local metric — no API key needed +result = evaluate("contains", output="Hello world", keyword="Hello") +print(result.score) # 1.0 +print(result.passed) # True + +# Cloud metric — needs FI_API_KEY and FI_SECRET_KEY +result = evaluate("toxicity", output="Hello world", model="turing_flash") +print(result.score) # 1.0 +print(result.passed) # True +``` + +## Optional extras (Python) + +| Extra | Install | What it adds | +|-------|---------|-------------| +| NLI models | `pip install ai-evaluation[nli]` | DeBERTa for faithfulness and hallucination detection | +| Embeddings | `pip install ai-evaluation[embeddings]` | Sentence-transformers for semantic similarity | +| Feedback | `pip install ai-evaluation[feedback]` | ChromaDB-backed feedback collection | +| Distributed | `pip install ai-evaluation[celery]` | Celery + Redis for distributed eval runs | +| Everything | `pip install ai-evaluation[all]` | All optional dependencies | + +## Full reference + + + + All 76+ metrics, engine routing, LLM-as-Judge, streaming, and distributed eval. + + + Real-time guardrails for toxicity, PII, and prompt injection. + + diff --git a/src/pages/docs/sdk/list/traceai.mdx b/src/pages/docs/sdk/list/traceai.mdx new file mode 100644 index 00000000..4be917b6 --- /dev/null +++ b/src/pages/docs/sdk/list/traceai.mdx @@ -0,0 +1,103 @@ +--- +title: "traceAI" +description: "OpenTelemetry tracing for AI apps. Auto-instrument 45+ frameworks or add custom spans with fi-instrumentation-otel." +--- + +Trace LLM calls, retrieval steps, and agent actions with `fi-instrumentation-otel` plus one `traceai-*` package per framework. Call `register()` once, then auto-instrument your stack. Available in Python, TypeScript, Java, and C#. + +```bash +pip install fi-instrumentation-otel traceai-openai +``` + +```python +from fi_instrumentation import register +from fi_instrumentation.fi_types import ProjectType + +trace_provider = register( + project_name="my-project", + project_type=ProjectType.OBSERVE, +) + +from traceai_openai import OpenAIInstrumentor +OpenAIInstrumentor().instrument(tracer_provider=trace_provider) + +# All OpenAI calls are now traced +# Traces appear in your Future AGI dashboard under "my-project" +``` + +## Supported instrumentors + +Each instrumentor is lightweight and independent. Install only the ones for frameworks you actually use. For per-framework setup, see [Integrations](/docs/integrations). + + + + | Package | Framework | + |---------|-----------| + | `traceai-openai` | OpenAI | + | `traceai-anthropic` | Anthropic | + | `traceai-google-genai` | Google Generative AI | + | `traceai-vertexai` | Google Vertex AI | + | `traceai-bedrock` | AWS Bedrock | + | `traceai-mistralai` | Mistral AI | + | `traceai-groq` | Groq | + | `traceai-litellm` | LiteLLM | + | `traceai-cohere` | Cohere | + | `traceai-ollama` | Ollama | + | `traceai-deepseek` | DeepSeek | + | `traceai-together` | Together AI | + | `traceai-fireworks` | Fireworks AI | + | `traceai-cerebras` | Cerebras | + | `traceai-xai` | xAI / Grok | + | `traceai-vllm` | vLLM | + | `traceai-portkey` | Portkey | + | `traceai-huggingface` | HuggingFace | + + + | Package | Framework | + |---------|-----------| + | `traceai-langchain` | LangChain / LangGraph | + | `traceai-llamaindex` | LlamaIndex | + | `traceai-crewai` | CrewAI | + | `traceai-openai-agents` | OpenAI Agents SDK | + | `traceai-autogen` | Microsoft AutoGen | + | `traceai-smolagents` | HuggingFace SmolAgents | + | `traceai-google-adk` | Google Agent Dev Kit | + | `traceai-claude-agent-sdk` | Claude Agent SDK | + | `traceai-pydantic-ai` | Pydantic AI | + | `traceai-strands` | AWS Strands Agents | + | `traceai-agno` | Agno | + | `traceai-beeai` | IBM BeeAI | + | `traceai-haystack` | Haystack | + | `traceai-dspy` | DSPy | + | `traceai-guardrails` | Guardrails AI | + | `traceai-instructor` | Instructor | + | `traceai-mcp` | Model Context Protocol | + + + | Package | Framework | + |---------|-----------| + | `traceai-pipecat` | Pipecat | + | `traceai-livekit` | LiveKit | + + + | Package | Framework | + |---------|-----------| + | `traceai-pinecone` | Pinecone | + | `traceai-chromadb` | ChromaDB | + | `traceai-qdrant` | Qdrant | + | `traceai-weaviate` | Weaviate | + | `traceai-milvus` | Milvus | + | `traceai-lancedb` | LanceDB | + | `traceai-mongodb` | MongoDB | + | `traceai-pgvector` | pgvector | + | `traceai-redis` | Redis | + + + +## Full reference + + + + register(), FITracer, context helpers, how-to guides, and semantic conventions. + + diff --git a/src/pages/docs/sdk/tracing.mdx b/src/pages/docs/sdk/tracing.mdx deleted file mode 100644 index cdcb2ef2..00000000 --- a/src/pages/docs/sdk/tracing.mdx +++ /dev/null @@ -1,1257 +0,0 @@ ---- -title: "Tracing: OpenTelemetry SDK for Future AGI AI Apps" -description: "Set up OpenTelemetry tracing across Python, TypeScript, Java, and C#. Auto-instrument 45+ frameworks or create custom spans with FITracer." ---- - - -- `register()` sets up the tracer provider in two lines, all languages -- Auto-instrument with `traceai-*` packages (45+ frameworks) or create custom spans with `FITracer` -- Context helpers attach session, user, metadata, and tags to all spans in a block -- TraceConfig controls privacy masking, PII redaction covers 6 data types automatically - - -The pattern is the same across all four languages: call `register()` once to set up the provider, then either auto-instrument your frameworks or use `FITracer` for custom spans. LLM calls, retrieval steps, and agent actions get captured as OpenTelemetry spans and sent to your dashboard. - - - Requires `FI_API_KEY` and `FI_SECRET_KEY` in your environment. For conceptual background on traces, spans, and attributes, see the [Tracing guide](/docs/tracing/auto). - - -## Quick Example - - - - ```bash - pip install fi-instrumentation-otel traceai-openai - ``` - - ```python - from fi_instrumentation import register - from fi_instrumentation.fi_types import ProjectType - from traceai_openai import OpenAIInstrumentor - - # 1. Register the tracer provider - trace_provider = register( - project_name="my-project", - project_type=ProjectType.OBSERVE, - ) - - # 2. Instrument your framework - OpenAIInstrumentor().instrument(tracer_provider=trace_provider) - - # 3. Use OpenAI as normal - all calls are now traced - import openai - client = openai.OpenAI() - response = client.chat.completions.create( - model="gpt-4o-mini", - messages=[{"role": "user", "content": "What is Python?"}], - ) - ``` - - - ```bash - npm install @traceai/openai @traceai/fi-core @opentelemetry/instrumentation - ``` - - ```typescript - import { register, ProjectType } from "@traceai/fi-core"; - import { OpenAIInstrumentation } from "@traceai/openai"; - import { registerInstrumentations } from "@opentelemetry/instrumentation"; - import OpenAI from "openai"; - - const tracerProvider = register({ - projectName: "my-project", - projectType: ProjectType.OBSERVE, - }); - - registerInstrumentations({ - tracerProvider, - instrumentations: [new OpenAIInstrumentation()], - }); - - const openai = new OpenAI(); - const response = await openai.chat.completions.create({ - model: "gpt-4o-mini", - messages: [{ role: "user", content: "Hello!" }], - }); - ``` - - - ```xml - - - com.github.future-agi.traceAI - traceai-spring-boot-starter - v1.0.0 - - - com.github.future-agi.traceAI - traceai-java-openai - v1.0.0 - - ``` - - ```java - import ai.traceai.TraceAI; - import ai.traceai.TraceConfig; - import ai.traceai.openai.TracedOpenAIClient; - - // Initialize from environment variables - TraceAI.initFromEnvironment(); - - // Wrap your client - TracedOpenAIClient tracedClient = new TracedOpenAIClient(openAIClient); - var response = tracedClient.createChatCompletion(params); - ``` - - Set `FI_API_KEY`, `FI_SECRET_KEY`, `FI_BASE_URL`, and `FI_PROJECT_NAME` as environment variables. - - - ```bash - dotnet add package fi-instrumentation-otel - ``` - - ```csharp - using FIInstrumentation; - using FIInstrumentation.Types; - - var tracer = TraceAI.Register(opts => - { - opts.ProjectName = "my-project"; - opts.ProjectType = ProjectType.Observe; - }); - - // Create traced LLM calls with convenience methods - var result = tracer.Llm("openai-call", span => - { - span.SetInput("What is C#?"); - var response = CallOpenAI("What is C#?"); - span.SetOutput(response); - return response; - }); - - TraceAI.Shutdown(); - ``` - - - -## register() - -Creates an OpenTelemetry tracer provider configured to export spans to your Future AGI dashboard. - - - - ```python - from fi_instrumentation import register - from fi_instrumentation.fi_types import ProjectType, Transport - - trace_provider = register( - project_name="my-project", - project_type=ProjectType.OBSERVE, - transport=Transport.HTTP, - batch=True, - verbose=True, - ) - ``` - - | Parameter | Type | Default | Description | - |-----------|------|---------|-------------| - | `project_name` | str / None | `FI_PROJECT_NAME` env var | Project identifier in the dashboard | - | `project_type` | ProjectType | `EXPERIMENT` | `EXPERIMENT` (dev, supports eval tags) or `OBSERVE` (production) | - | `project_version_name` | str / None | None | Version label (EXPERIMENT only) | - | `eval_tags` | list / None | None | Evaluation configs for automated span scoring (EXPERIMENT only) | - | `metadata` | dict / None | None | Custom metadata attached to all spans | - | `batch` | bool | True | True = BatchSpanProcessor, False = SimpleSpanProcessor | - | `set_global_tracer_provider` | bool | False | Register as the global OpenTelemetry default | - | `headers` | dict / None | None | Custom HTTP headers (auto-populated from API keys if not set) | - | `verbose` | bool | True | Print configuration details on startup | - | `transport` | Transport | `HTTP` | `HTTP` or `GRPC` | - | `semantic_convention` | SemanticConvention | `FI` | Attribute naming convention | - - **Returns:** `TracerProvider` - pass this to `.instrument(tracer_provider=...)` on any instrumentor. - - - ```typescript - import { register, ProjectType, Transport } from "@traceai/fi-core"; - - const tracerProvider = register({ - projectName: "my-project", - projectType: ProjectType.OBSERVE, - transport: Transport.HTTP, - batch: true, - verbose: true, - }); - ``` - - | Parameter | Type | Default | Description | - |-----------|------|---------|-------------| - | `projectName` | string | `FI_PROJECT_NAME` env var | Project identifier | - | `projectType` | ProjectType | `EXPERIMENT` | `EXPERIMENT` or `OBSERVE` | - | `projectVersionName` | string | undefined | Version label (EXPERIMENT only) | - | `evalTags` | EvalTag[] | undefined | Evaluation configs (EXPERIMENT only) | - | `sessionName` | string | undefined | Session name (OBSERVE only) | - | `metadata` | Record | undefined | Custom metadata | - | `batch` | boolean | false | Use batch span processor | - | `setGlobalTracerProvider` | boolean | true | Register as global provider | - | `headers` | FIHeaders | undefined | Custom HTTP headers | - | `verbose` | boolean | false | Verbose logging | - | `endpoint` | string | `FI_BASE_URL` | Custom endpoint | - | `transport` | Transport | `HTTP` | `HTTP` or `GRPC` | - - **Returns:** `FITracerProvider` - - - ```java - import ai.traceai.TraceAI; - import ai.traceai.TraceConfig; - - // Option 1: From environment variables - TraceAI.initFromEnvironment(); - - // Option 2: Programmatic configuration - TraceAI.init(TraceConfig.builder() - .baseUrl("https://api.futureagi.com") - .apiKey("your-api-key") - .secretKey("your-secret-key") - .projectName("my-project") - .batchSize(512) - .exportIntervalMs(5000) - .build() - ); - - FITracer tracer = TraceAI.getTracer(); - ``` - - | Builder method | Default | Description | - |----------------|---------|-------------| - | `baseUrl(String)` | `FI_BASE_URL` env var | Backend endpoint | - | `apiKey(String)` | `FI_API_KEY` env var | API authentication | - | `secretKey(String)` | `FI_SECRET_KEY` env var | Secondary authentication | - | `projectName(String)` | `FI_PROJECT_NAME` env var | Project identifier | - | `serviceName(String)` | project name | OpenTelemetry service name | - | `hideInputs(boolean)` | false | Suppress input values | - | `hideOutputs(boolean)` | false | Suppress output values | - | `hideInputMessages(boolean)` | false | Suppress input messages | - | `hideOutputMessages(boolean)` | false | Suppress output messages | - | `enableConsoleExporter(boolean)` | false | Log spans to console | - | `batchSize(int)` | 512 | Span batch size | - | `exportIntervalMs(long)` | 5000 | Export interval in ms | - - For **Spring Boot**, add the starter dependency and configure via `application.yml`: - - ```yaml - traceai: - enabled: true - base-url: https://api.futureagi.com - api-key: ${FI_API_KEY} - secret-key: ${FI_SECRET_KEY} - project-name: my-app - batch-size: 512 - export-interval-ms: 5000 - ``` - - The `FITracer` bean is auto-created and available for injection. - - - ```csharp - using FIInstrumentation; - using FIInstrumentation.Types; - - var tracer = TraceAI.Register(opts => - { - opts.ProjectName = "my-project"; - opts.ProjectType = ProjectType.Observe; - opts.Transport = Transport.Http; - opts.Batch = true; - opts.Verbose = true; - opts.TraceConfig = TraceConfig.Builder() - .HideInputs(false) - .HideOutputs(false) - .Build(); - }); - ``` - - | Property | Type | Default | Description | - |----------|------|---------|-------------| - | `ProjectName` | string | `FI_PROJECT_NAME` env var | Project identifier | - | `ProjectType` | ProjectType | Experiment | `Experiment` or `Observe` | - | `ProjectVersionName` | string | null | Version label (Experiment only) | - | `EvalTags` | List<EvalTag> | null | Evaluation configs (Experiment only) | - | `Metadata` | Dictionary | null | Custom metadata | - | `Batch` | bool | true | Use batch span processor | - | `SetGlobalTracerProvider` | bool | true | Register as global provider | - | `Transport` | Transport | Http | `Http` or `Grpc` | - | `ApiKey` | string | `FI_API_KEY` env var | API key | - | `SecretKey` | string | `FI_SECRET_KEY` env var | Secret key | - | `TraceConfig` | TraceConfig | null | Privacy/masking configuration | - | `EnableConsoleExporter` | bool | false | Log spans to console | - | `Verbose` | bool | true | Print config on startup | - - **Returns:** `FITracer` - use for creating custom spans. - - - -### ProjectType - -| Value | Use for | -|-------|---------| -| `EXPERIMENT` | Development and testing. Supports eval tags and version names. | -| `OBSERVE` | Production monitoring. No eval tags, no version names. | - -### SemanticConvention (Python/TypeScript) - -Controls how span attributes are named. We recommend `OTEL_GENAI` for standard OpenTelemetry GenAI conventions. - -| Value | Attribute prefix | Use for | -|-------|-----------------|---------| -| `OTEL_GENAI` | `gen_ai.*` | Recommended - OpenTelemetry GenAI standard | -| `FI` | `fi.*` | Legacy Future AGI format (default) | -| `OPENINFERENCE` | `openinference.*` | Arize Phoenix compatibility | -| `OPENLLMETRY` | `traceloop.*` | Traceloop / OpenLLMetry compatibility | - - - Pass `semantic_convention=SemanticConvention.OTEL_GENAI` for the best interoperability with other OpenTelemetry tools. - - -## FITracer - Custom Spans - -Beyond auto-instrumentation, `FITracer` lets you create custom spans for your own logic - agent steps, chain stages, tool calls, or any operation you want to trace. - -### Span Kinds - -All languages share the same span kinds: - -| Kind | Use for | -|------|---------| -| `LLM` | Language model inference calls | -| `CHAIN` | Sequential pipeline steps | -| `AGENT` | Autonomous agent actions | -| `TOOL` | Tool/function calls | -| `EMBEDDING` | Vector generation | -| `RETRIEVER` | Document retrieval (RAG) | -| `RERANKER` | Re-ranking operations | -| `GUARDRAIL` | Safety/validation checks | -| `EVALUATOR` | Quality scoring | -| `UNKNOWN` | Unspecified or unexpected span type | -| `WORKFLOW` | Custom pipeline steps (Java only) | -| `CONVERSATION` | Voice/conversational AI (Java/C#) | -| `VECTOR_DB` | Vector database operations (Java/C#) | - -### Decorators and Convenience Methods - - - - Python's `FITracer` provides decorators for clean span creation: - - ```python - from fi_instrumentation import register - from fi_instrumentation.fi_types import ProjectType - - trace_provider = register( - project_name="my-project", - project_type=ProjectType.OBSERVE, - ) - tracer = trace_provider.get_tracer(__name__) - - # Use the FITracer wrapper for decorators - from fi_instrumentation import FITracer - fi_tracer = FITracer(tracer) - - @fi_tracer.agent(name="research-agent") - def research_agent(query): - # This entire function becomes an AGENT span - results = search(query) - return summarize(results) - - @fi_tracer.chain(name="rag-pipeline") - def rag_pipeline(question): - docs = retrieve(question) - return generate(question, docs) - - @fi_tracer.tool( - name="web-search", - description="Searches the web", - parameters={"query": {"type": "string"}} - ) - def web_search(query): - return requests.get(f"https://api.search.com?q={query}").json() - ``` - - You can also use context managers for manual span creation: - - ```python - from fi_instrumentation.fi_types import FiSpanKindValues - - with fi_tracer.start_as_current_span( - "llm-call", - fi_span_kind=FiSpanKindValues.LLM, - ) as span: - span.set_input(value="What is Python?") - response = call_llm("What is Python?") - span.set_output(value=response) - span.set_attributes({ - "gen_ai.request.model": "gpt-4o", - "gen_ai.usage.input_tokens": 10, - "gen_ai.usage.output_tokens": 150, - }) - ``` - - - TypeScript uses OpenTelemetry's standard `startActiveSpan` pattern: - - ```typescript - import { trace } from "@opentelemetry/api"; - - const tracer = trace.getTracer("my-app"); - - // Manual span creation - tracer.startActiveSpan("rag-pipeline", (span) => { - span.setAttribute("gen_ai.span.kind", "CHAIN"); - span.setAttribute("input.value", question); - - const docs = retrieve(question); - const result = generate(question, docs); - - span.setAttribute("output.value", result); - span.end(); - return result; - }); - ``` - - Context management functions let you set session, user, and metadata: - - ```typescript - import { - setSession, setUser, setMetadata, setTags, - getAttributesFromContext - } from "@traceai/fi-core"; - import { context } from "@opentelemetry/api"; - - const ctx = setSession(context.active(), { sessionId: "sess-123" }); - const ctx2 = setUser(ctx, { userId: "user-456" }); - - context.with(ctx2, () => { - // All spans created here inherit session and user - tracer.startActiveSpan("operation", (span) => { - // span automatically gets session.id and user.id - span.end(); - }); - }); - ``` - - - Java offers both lambda-based and manual span creation: - - ```java - import ai.traceai.FITracer; - import ai.traceai.FISpanKind; - - FITracer tracer = TraceAI.getTracer(); - - // Lambda-based - auto-manages span lifecycle - String result = tracer.trace("rag-pipeline", FISpanKind.CHAIN, (span) -> { - tracer.setInputValue(span, question); - - String docs = tracer.trace("retrieve", FISpanKind.RETRIEVER, (rSpan) -> { - tracer.setInputValue(rSpan, question); - var retrieved = vectorDb.search(question); - tracer.setOutputValue(rSpan, tracer.toJson(retrieved)); - return retrieved; - }); - - String answer = tracer.trace("generate", FISpanKind.LLM, (lSpan) -> { - tracer.setInputMessages(lSpan, List.of( - tracer.message("system", "Answer using the context."), - tracer.message("user", question) - )); - var resp = llm.generate(question, docs); - tracer.setOutputMessages(lSpan, List.of( - tracer.message("assistant", resp) - )); - tracer.setTokenCounts(lSpan, 50, 200, 250); - return resp; - }); - - tracer.setOutputValue(span, answer); - return answer; - }); - ``` - - Manual span creation for more control: - - ```java - import io.opentelemetry.api.trace.Span; - import io.opentelemetry.context.Context; - - Span span = tracer.startSpan("tool-call", FISpanKind.TOOL); - try { - tracer.setInputValue(span, inputJson); - String result = executeTool(inputJson); - tracer.setOutputValue(span, result); - span.setStatus(StatusCode.OK); - } catch (Exception e) { - tracer.setError(span, e); - } finally { - span.end(); - } - ``` - - - C# provides typed convenience methods for each span kind: - - ```csharp - var tracer = TraceAI.Register(opts => - { - opts.ProjectName = "my-project"; - opts.ProjectType = ProjectType.Observe; - }); - - // Convenience methods for each span kind - var result = tracer.Chain("rag-pipeline", span => - { - span.SetInput("What is quantum computing?"); - - var docs = tracer.Tool("vector-search", toolSpan => - { - toolSpan.SetTool("search", "Searches vector DB"); - toolSpan.SetInput("quantum computing"); - var results = vectorDb.Search("quantum computing"); - toolSpan.SetOutput(results); - return results; - }); - - var answer = tracer.Llm("generate", llmSpan => - { - llmSpan.SetAttribute(SemanticConventions.GenAiRequestModel, "gpt-4o"); - llmSpan.SetInputMessages(new List> - { - FITracer.Message("user", "What is quantum computing?") - }); - var resp = llm.Generate("What is quantum computing?", docs); - llmSpan.SetOutputMessages(new List> - { - FITracer.Message("assistant", resp) - }); - llmSpan.SetTokenCounts(50, 200, 250); - return resp; - }); - - span.SetOutput(answer); - return answer; - }); - - // Async variants - await tracer.AgentAsync("research-agent", async span => - { - span.SetInput("Research topic X"); - var result = await RunResearchAsync("topic X"); - span.SetOutput(result); - }); - ``` - - Manual span creation: - - ```csharp - using var span = tracer.StartSpan("custom-op", FISpanKind.Chain); - span.SetInput("input data"); - span.SetOutput("output data"); - // span.Dispose() ends the span automatically - ``` - - - -### FISpan Methods - -All languages provide methods on the span object for setting structured data: - -| Method | Description | Available in | -|--------|-------------|-------------| -| `set_input(value, mime_type=)` / `SetInput(value, mimeType)` | Set span input value (text or JSON). `mime_type` accepts `"text/plain"` or `"application/json"` | Python, C# | -| `set_output(value, mime_type=)` / `SetOutput(value, mimeType)` | Set span output value | Python, C# | -| `set_tool(name, description, parameters)` / `SetTool(...)` | Attach tool metadata | Python, C# | -| `set_attributes(dict)` / `SetAttribute(key, value)` | Set custom attributes | All | -| `setInputValue(span, value)` | Set input on span | Java | -| `setOutputValue(span, value)` | Set output on span | Java | -| `setInputMessages(span, messages)` / `SetInputMessages(messages)` | Set chat message history | Java, C# | -| `setOutputMessages(span, messages)` / `SetOutputMessages(messages)` | Set response messages | Java, C# | -| `setTokenCounts(span, in, out, total)` / `SetTokenCounts(in, out, total)` | Set token usage | Java, C# | -| `setError(span, exception)` / `SetError(exception)` | Record an exception | Java, C# | - - - In Java, these methods live on `FITracer` and take the span as the first argument (e.g. `tracer.setInputValue(span, value)`). In Python and C#, they're called directly on the span object. - - -## Context Helpers - -Attach metadata, tags, session IDs, and user IDs to spans. These apply to all spans created within the scope. - - - - ```python - from fi_instrumentation import ( - using_session, using_user, using_metadata, - using_tags, using_prompt_template, using_attributes, - suppress_tracing - ) - - # Individual context managers - with using_session("session-abc-123"): - with using_user("user-456"): - response = client.chat.completions.create(...) - - with using_metadata({"environment": "production", "version": "2.1"}): - response = client.chat.completions.create(...) - - with using_tags(["rag-pipeline", "v2"]): - response = client.chat.completions.create(...) - - # Prompt template tracking - with using_prompt_template( - template="Answer {question} using {context}", - label="production", - version="v1.2", - variables={"question": "...", "context": "..."} - ): - response = client.chat.completions.create(...) - - # Combined - set everything at once - with using_attributes( - session_id="session-abc", - user_id="user-456", - metadata={"env": "prod"}, - tags=["rag", "v2"], - prompt_template="Answer {question}", - prompt_template_version="v1.2", - ): - response = client.chat.completions.create(...) - - # Suppress tracing for a block - with suppress_tracing(): - # These calls won't be traced - result = client.chat.completions.create(...) - ``` - - - ```typescript - import { - setSession, getSession, clearSession, - setUser, getUser, clearUser, - setMetadata, setTags, - setPromptTemplate, - getAttributesFromContext - } from "@traceai/fi-core"; - import { context } from "@opentelemetry/api"; - - // Build up context with multiple attributes - let ctx = context.active(); - ctx = setSession(ctx, { sessionId: "session-abc-123" }); - ctx = setUser(ctx, { userId: "user-456" }); - ctx = setMetadata(ctx, { environment: "production" }); - ctx = setTags(ctx, ["rag-pipeline", "v2"]); - ctx = setPromptTemplate(ctx, { - template: "Answer {{question}} using {{context}}", - variables: { question: "...", context: "..." }, - version: "v1.2", - }); - - // All spans created in this context inherit these attributes - context.with(ctx, async () => { - const response = await openai.chat.completions.create({ - model: "gpt-4o-mini", - messages: [{ role: "user", content: "Hello" }], - }); - }); - - // Read attributes back from context - const attrs = getAttributesFromContext(ctx); - ``` - - - Java uses `AutoCloseable` scopes with try-with-resources: - - ```java - import ai.traceai.ContextAttributes; - - // Session tracking - try (var ignored = ContextAttributes.usingSession("session-abc-123")) { - // All spans here get session.id and gen_ai.conversation.id - var response = tracedClient.createChatCompletion(params); - } - - // User tracking - try (var ignored = ContextAttributes.usingUser("user-456")) { - var response = tracedClient.createChatCompletion(params); - } - - // Metadata - try (var ignored = ContextAttributes.usingMetadata(Map.of( - "environment", "production", - "version", "2.1" - ))) { - var response = tracedClient.createChatCompletion(params); - } - - // Tags - try (var ignored = ContextAttributes.usingTags(List.of("rag-pipeline", "v2"))) { - var response = tracedClient.createChatCompletion(params); - } - - // Nest them for combined context - try (var s = ContextAttributes.usingSession("session-abc"); - var u = ContextAttributes.usingUser("user-456"); - var m = ContextAttributes.usingMetadata(Map.of("env", "prod"))) { - var response = tracedClient.createChatCompletion(params); - } - - // Read current attributes - Map attrs = ContextAttributes.getAttributesFromContext(); - ``` - - - C# uses `IDisposable` scopes with `using` statements: - - ```csharp - using FIInstrumentation.Context; - - // Session and user tracking - using (ContextAttributes.UsingSession("session-abc-123")) - using (ContextAttributes.UsingUser("user-456")) - { - tracer.Llm("llm-call", span => - { - // span automatically gets session.id and user.id - span.SetInput("Hello!"); - }); - } - - // Metadata and tags - using (ContextAttributes.UsingMetadata(new Dictionary - { - ["environment"] = "production", - ["version"] = "2.1" - })) - using (ContextAttributes.UsingTags(new List { "rag-pipeline", "v2" })) - { - tracer.Chain("pipeline", span => { /* ... */ }); - } - - // Prompt template tracking - using (ContextAttributes.UsingPromptTemplate( - template: "Answer {question} using {context}", - label: "production", - version: "v1.2", - variables: new Dictionary - { - ["question"] = "...", - ["context"] = "..." - } - )) - { - tracer.Llm("templated-call", span => { /* ... */ }); - } - - // Combined - set everything at once - using (ContextAttributes.UsingAttributes( - sessionId: "session-abc", - userId: "user-456", - metadata: new Dictionary { ["env"] = "prod" }, - tags: new List { "rag", "v2" } - )) - { - tracer.Chain("full-context", span => { /* ... */ }); - } - ``` - - - -### Suppress Tracing - -Temporarily disable tracing for a block of code. Useful for health checks, internal calls, or operations you don't want in your traces. Available in Python and C# only - Java and TypeScript don't have this API. - - - - ```python - from fi_instrumentation import suppress_tracing - - with suppress_tracing(): - # Nothing in this block is traced - result = client.chat.completions.create(...) - ``` - - - ```csharp - using FIInstrumentation.Context; - - using (new SuppressTracing()) - { - // Nothing in this block is traced - } - ``` - - - -## TraceConfig - -Control what data gets captured. Useful for privacy compliance, reducing payload size, or masking sensitive data. - - - - ```python - from fi_instrumentation import TraceConfig - - config = TraceConfig( - hide_inputs=True, - hide_outputs=True, - pii_redaction=True, - ) - - # Pass to instrumentors - OpenAIInstrumentor().instrument( - tracer_provider=trace_provider, - config=config, - ) - ``` - - - ```java - TraceAI.init(TraceConfig.builder() - .baseUrl("https://api.futureagi.com") - .apiKey("your-key") - .projectName("my-project") - .hideInputs(true) - .hideOutputs(true) - .hideInputMessages(true) - .hideOutputMessages(true) - .build() - ); - ``` - - - In TypeScript, `TraceConfig` is passed per-instrumentor, not to `register()`: - - ```typescript - import { OpenAIInstrumentation } from "@traceai/openai"; - import { registerInstrumentations } from "@opentelemetry/instrumentation"; - - registerInstrumentations({ - tracerProvider, - instrumentations: [ - new OpenAIInstrumentation({ - traceConfig: { - hideInputs: true, - hideOutputs: true, - hideInputImages: true, - hideEmbeddingVectors: true, - base64ImageMaxLength: 16000, - piiRedaction: true, - }, - }), - ], - }); - ``` - - - ```csharp - var tracer = TraceAI.Register(opts => - { - opts.ProjectName = "my-project"; - opts.TraceConfig = TraceConfig.Builder() - .HideInputs(true) - .HideOutputs(true) - .HideInputImages(true) - .HideEmbeddingVectors(true) - .Base64ImageMaxLength(16000) - .Build(); - }); - ``` - - - -| Field | Type | Default | What it hides | -|-------|------|---------|--------------| -| `hide_inputs` | bool | False | All input values and messages | -| `hide_outputs` | bool | False | All output values and messages | -| `hide_input_messages` | bool | False | Input messages only | -| `hide_output_messages` | bool | False | Output messages only | -| `hide_input_images` | bool | False | Images in inputs | -| `hide_input_text` | bool | False | Text in input messages | -| `hide_output_text` | bool | False | Text in output messages | -| `hide_embedding_vectors` | bool | False | Embedding vectors | -| `hide_llm_invocation_parameters` | bool | False | Model parameters (temperature, etc.) | -| `base64_image_max_length` | int | 32000 | Truncate base64 images beyond this length | -| `pii_redaction` | bool | False | Automatically mask PII (Python only) | - -Each field maps to an environment variable with the `FI_` prefix (e.g. `hide_inputs` -> `FI_HIDE_INPUTS`). - -### PII Redaction (Python) - -When `pii_redaction=True`, the SDK automatically detects and masks 6 types of personally identifiable information: - -| PII Type | Pattern | Replaced with | -|----------|---------|--------------| -| Email addresses | `user@example.com` | `` | -| Social Security Numbers | `123-45-6789` | `` | -| Credit card numbers | `4111-1111-1111-1111` | `` | -| API keys | `sk_live_...`, `pk_test_...` | `` | -| IP addresses (IPv4) | `192.168.1.1` | `` | -| Phone numbers | `+1-555-123-4567` | `` | - -```python -# Enable via code -config = TraceConfig(pii_redaction=True) - -# Or via environment variable -# export FI_PII_REDACTION=true - -# Direct usage -from fi_instrumentation.instrumentation.pii_redaction import redact_pii_in_string - -redacted = redact_pii_in_string("Email me at test@example.com") -# "Email me at " -``` - -## EvalTags - Attach Evaluations to Traces - -EvalTags let you configure automatic evaluations that run server-side on your traced spans. Attach them during `register()` and the platform scores spans as they arrive. - - - - ```python - from fi_instrumentation import register - from fi_instrumentation.fi_types import ( - ProjectType, EvalTag, EvalTagType, - EvalSpanKind, EvalName, ModelChoices - ) - - trace_provider = register( - project_name="my-project", - project_type=ProjectType.EXPERIMENT, - project_version_name="v1.0", - eval_tags=[ - EvalTag( - type=EvalTagType.OBSERVATION_SPAN, - value=EvalSpanKind.LLM, - eval_name=EvalName.GROUNDEDNESS, - model=ModelChoices.TURING_FLASH, - ), - EvalTag( - type=EvalTagType.OBSERVATION_SPAN, - value=EvalSpanKind.LLM, - eval_name=EvalName.TOXICITY, - model=ModelChoices.TURING_FLASH, - ), - ], - ) - ``` - - - ```typescript - import { - register, ProjectType, EvalTag, - EvalTagType, EvalSpanKind, EvalName, ModelChoices - } from "@traceai/fi-core"; - - const tracerProvider = register({ - projectName: "my-project", - projectType: ProjectType.EXPERIMENT, - projectVersionName: "v1.0", - evalTags: [ - await EvalTag.create({ - type: EvalTagType.OBSERVATION_SPAN, - value: EvalSpanKind.LLM, - eval_name: EvalName.GROUNDEDNESS, - model: ModelChoices.TURING_FLASH, - }), - await EvalTag.create({ - type: EvalTagType.OBSERVATION_SPAN, - value: EvalSpanKind.LLM, - eval_name: EvalName.TOXICITY, - model: ModelChoices.TURING_FLASH, - }), - ], - }); - ``` - - - `EvalTag.create()` is async in TypeScript because it validates the eval configuration with the server. - - - - ```csharp - using FIInstrumentation; - using FIInstrumentation.Types; - - var tracer = TraceAI.Register(opts => - { - opts.ProjectName = "my-project"; - opts.ProjectType = ProjectType.Experiment; - opts.ProjectVersionName = "v1.0"; - opts.EvalTags = new List - { - new EvalTag(EvalSpanKind.Llm, EvalName.Groundedness) - { - Model = ModelChoices.TuringFlash, - }, - new EvalTag(EvalSpanKind.Llm, EvalName.Toxicity) - { - Model = ModelChoices.TuringFlash, - }, - }; - }); - ``` - - - -### EvalSpanKind - -Which span types to evaluate: - -| Value | Description | -|-------|-------------| -| `LLM` | Language model calls | -| `RETRIEVER` | Document retrieval spans | -| `TOOL` | Tool/function calls | -| `AGENT` | Agent spans | -| `EMBEDDING` | Embedding generation | -| `RERANKER` | Re-ranking operations | - -### ModelChoices - -Which evaluation model to use: - -| Value | Description | -|-------|-------------| -| `TURING_FLASH` | Fast evaluation model | -| `TURING_SMALL` | Small evaluation model | -| `TURING_LARGE` | High-accuracy evaluation model | -| `PROTECT` | Safety-focused model | -| `PROTECT_FLASH` | Fast safety model | - - - EvalTags only work with `ProjectType.EXPERIMENT`. For production monitoring without evals, use `ProjectType.OBSERVE`. - - -## Instrumentors - -Each framework has its own instrumentor package. Install the one for your framework and call `.instrument()`. - -```python -# Pattern is the same for every framework: -from traceai_ import Instrumentor -Instrumentor().instrument(tracer_provider=trace_provider) -``` - - - - | Package | Framework | Instrumentor class | - |---------|-----------|-------------------| - | `traceai-openai` | OpenAI | `OpenAIInstrumentor` | - | `traceai-anthropic` | Anthropic | `AnthropicInstrumentor` | - | `traceai-google-genai` | Google GenAI | `GoogleGenAIInstrumentor` | - | `traceai-vertexai` | Vertex AI | `VertexAIInstrumentor` | - | `traceai-bedrock` | AWS Bedrock | `BedrockInstrumentor` | - | `traceai-mistralai` | Mistral AI | `MistralAIInstrumentor` | - | `traceai-groq` | Groq | `GroqInstrumentor` | - | `traceai-litellm` | LiteLLM | `LiteLLMInstrumentor` | - | `traceai-cohere` | Cohere | `CohereInstrumentor` | - | `traceai-ollama` | Ollama | `OllamaInstrumentor` | - | `traceai-deepseek` | DeepSeek | `DeepSeekInstrumentor` | - | `traceai-together` | Together AI | `TogetherInstrumentor` | - | `traceai-fireworks` | Fireworks AI | `FireworksInstrumentor` | - | `traceai-cerebras` | Cerebras | `CerebrasInstrumentor` | - | `traceai-xai` | xAI / Grok | `XAIInstrumentor` | - | `traceai-vllm` | vLLM | `VLLMInstrumentor` | - | `traceai-portkey` | Portkey | `PortkeyInstrumentor` | - | `traceai-huggingface` | HuggingFace | `HuggingFaceInstrumentor` | - - - | Package | Framework | Instrumentor class | - |---------|-----------|-------------------| - | `traceai-langchain` | LangChain / LangGraph | `LangChainInstrumentor` | - | `traceai-llamaindex` | LlamaIndex | `LlamaIndexInstrumentor` | - | `traceai-crewai` | CrewAI | `CrewAIInstrumentor` | - | `traceai-openai-agents` | OpenAI Agents SDK | `OpenAIAgentsInstrumentor` | - | `traceai-autogen` | Microsoft AutoGen | `AutoGenInstrumentor` | - | `traceai-smolagents` | HuggingFace SmolAgents | `SmolAgentsInstrumentor` | - | `traceai-google-adk` | Google Agent Dev Kit | `GoogleADKInstrumentor` | - | `traceai-claude-agent-sdk` | Claude Agent SDK | `ClaudeAgentSDKInstrumentor` | - | `traceai-pydantic-ai` | Pydantic AI | `PydanticAIInstrumentor` | - | `traceai-strands` | AWS Strands Agents | `StrandsInstrumentor` | - | `traceai-agno` | Agno | `AgnoInstrumentor` | - | `traceai-beeai` | IBM BeeAI | `BeeAIInstrumentor` | - | `traceai-haystack` | Haystack | `HaystackInstrumentor` | - | `traceai-dspy` | DSPy | `DSPyInstrumentor` | - | `traceai-guardrails` | Guardrails AI | `GuardrailsInstrumentor` | - | `traceai-instructor` | Instructor | `InstructorInstrumentor` | - | `traceai-mcp` | Model Context Protocol | `MCPInstrumentor` | - - - | Package | Framework | Instrumentor class | - |---------|-----------|-------------------| - | `traceai-pipecat` | Pipecat | `PipecatInstrumentor` | - | `traceai-livekit` | LiveKit | `LiveKitInstrumentor` | - - - | Package | Framework | Instrumentor class | - |---------|-----------|-------------------| - | `traceai-pinecone` | Pinecone | `PineconeInstrumentor` | - | `traceai-chromadb` | ChromaDB | `ChromaDBInstrumentor` | - | `traceai-qdrant` | Qdrant | `QdrantInstrumentor` | - | `traceai-weaviate` | Weaviate | `WeaviateInstrumentor` | - | `traceai-milvus` | Milvus | `MilvusInstrumentor` | - | `traceai-lancedb` | LanceDB | `LanceDBInstrumentor` | - | `traceai-mongodb` | MongoDB | `MongoDBInstrumentor` | - | `traceai-pgvector` | pgvector | `PgVectorInstrumentor` | - | `traceai-redis` | Redis | `RedisInstrumentor` | - - - -### Cleanup - -To remove instrumentation (useful in tests or serverless cleanup): - - - - ```python - OpenAIInstrumentor().uninstrument() - ``` - - - ```java - TraceAI.shutdown(); // Flushes remaining spans and shuts down - ``` - - - ```csharp - TraceAI.Shutdown(); // Flushes remaining spans and shuts down - ``` - - - -For per-framework setup guides with full examples, see the [Auto-Instrumentation docs](/docs/tracing/auto). - -### Other Languages - -The tables above show Python packages. TypeScript, Java, and C# have their own instrumentation libraries: - - - - TypeScript packages follow the `@traceai/` pattern. All use OpenTelemetry's `registerInstrumentations()`. - - ```typescript - import { registerInstrumentations } from "@opentelemetry/instrumentation"; - import { OpenAIInstrumentation } from "@traceai/openai"; - import { AnthropicInstrumentation } from "@traceai/anthropic"; - import { LangChainInstrumentation } from "@traceai/langchain"; - import { PineconeInstrumentation } from "@traceai/pinecone"; - - registerInstrumentations({ - tracerProvider, - instrumentations: [ - new OpenAIInstrumentation(), - new AnthropicInstrumentation(), - new LangChainInstrumentation(), - new PineconeInstrumentation(), - ], - }); - ``` - - 40+ packages available including all LLM providers, frameworks, and vector DBs from the Python list, plus `@traceai/vercel` for Vercel/Next.js and `@traceai/mastra`. - - - Java uses the `Traced*` wrapper pattern. Each integration wraps the native client: - - ```java - // LLM Providers - TracedOpenAIClient traced = new TracedOpenAIClient(openAIClient); - TracedAnthropicClient traced = new TracedAnthropicClient(anthropicClient); - TracedBedrockRuntimeClient traced = new TracedBedrockRuntimeClient(bedrockClient); - TracedGenerativeModel traced = new TracedGenerativeModel(model); // Google GenAI - TracedOllamaAPI traced = new TracedOllamaAPI(ollamaAPI); - TracedCohereClient traced = new TracedCohereClient(cohereClient); - TracedWatsonxAI traced = new TracedWatsonxAI(watsonxClient); - - // Vector Databases - TracedPineconeIndex traced = new TracedPineconeIndex(index, "my-index"); - TracedQdrantClient traced = new TracedQdrantClient(qdrantClient); - TracedMilvusClient traced = new TracedMilvusClient(milvusClient); - TracedChromaCollection traced = new TracedChromaCollection(collection); - TracedMongoVectorSearch traced = new TracedMongoVectorSearch(collection); - TracedRedisVectorSearch traced = new TracedRedisVectorSearch(jedis); - TracedSearchClient traced = new TracedSearchClient(searchClient); // Azure Search - TracedPgVectorStore traced = new TracedPgVectorStore(connection); - TracedElasticsearchClient traced = new TracedElasticsearchClient(esClient); - - // Framework integrations - TracedChatLanguageModel traced = new TracedChatLanguageModel(model, tracer, "openai"); // LangChain4j - TracedChatModel traced = new TracedChatModel(chatModel, tracer, "openai"); // Spring AI - TracedKernel traced = new TracedKernel(kernel, tracer); // Semantic Kernel - ``` - - Maven coordinates: `com.github.future-agi.traceAI:traceai-java-:v1.0.0` - - - C# uses manual tracing via `FITracer`. No auto-instrumentation wrappers yet - use the convenience methods (`Llm()`, `Chain()`, `Agent()`, `Tool()`) to create spans around your calls. - - ```csharp - // Wrap any LLM call - var response = tracer.Llm("openai-call", span => - { - span.SetAttribute(SemanticConventions.GenAiRequestModel, "gpt-4o"); - span.SetInput(prompt); - var result = CallOpenAI(prompt); - span.SetOutput(result); - span.SetTokenCounts(inputTokens, outputTokens, totalTokens); - return result; - }); - ``` - - Install: `dotnet add package fi-instrumentation-otel` - - - -## Environment Variables - -All languages read from the same set of environment variables: - -| Variable | Purpose | Default | -|----------|---------|---------| -| `FI_API_KEY` | Authentication | required | -| `FI_SECRET_KEY` | Authentication | required | -| `FI_BASE_URL` | HTTP collector endpoint | `https://api.futureagi.com` | -| `FI_GRPC_URL` | gRPC collector endpoint | `https://grpc.futureagi.com` | -| `FI_PROJECT_NAME` | Default project name | None | -| `FI_PROJECT_VERSION_NAME` | Default version | None | -| `FI_HIDE_INPUTS` | Redact inputs | False | -| `FI_HIDE_OUTPUTS` | Redact outputs | False | -| `FI_HIDE_INPUT_MESSAGES` | Redact input messages | False | -| `FI_HIDE_OUTPUT_MESSAGES` | Redact output messages | False | -| `FI_HIDE_INPUT_IMAGES` | Redact input images | False | -| `FI_HIDE_INPUT_TEXT` | Redact input text | False | -| `FI_HIDE_OUTPUT_TEXT` | Redact output text | False | -| `FI_HIDE_EMBEDDING_VECTORS` | Redact embedding vectors | False | -| `FI_HIDE_LLM_INVOCATION_PARAMETERS` | Redact model parameters | False | -| `FI_BASE64_IMAGE_MAX_LENGTH` | Max base64 image chars | 32000 | -| `FI_PII_REDACTION` | Auto-mask PII (Python) | False | - -## Related - - - - Concepts, manual tracing, and per-framework setup guides. - - - Setup guides for all 45+ supported frameworks. - - - Score traced outputs with 76+ metrics. - - - Store test data and run batch evaluations. - - - Guard inputs and outputs with safety rules. - - - Test voice AI agents with simulated personas. - - diff --git a/src/pages/docs/observe/features/manual-tracing/add-attributes-metadata-tags.mdx b/src/pages/docs/sdk/tracing/add-attributes-metadata-tags.mdx similarity index 98% rename from src/pages/docs/observe/features/manual-tracing/add-attributes-metadata-tags.mdx rename to src/pages/docs/sdk/tracing/add-attributes-metadata-tags.mdx index 6ba8979d..c8b1803a 100644 --- a/src/pages/docs/observe/features/manual-tracing/add-attributes-metadata-tags.mdx +++ b/src/pages/docs/sdk/tracing/add-attributes-metadata-tags.mdx @@ -476,16 +476,16 @@ A trace with only timing and status tells what happened, but not why. Without at ## Next Steps - + Register a tracer provider and add instrumentation. - + Use FITracer decorators and context managers for typed spans. - + Group traces into sessions and link them to end users. - + Redact sensitive data with TraceConfig before export. diff --git a/src/pages/docs/observe/features/manual-tracing/add-events-exceptions-status.mdx b/src/pages/docs/sdk/tracing/add-events-exceptions-status.mdx similarity index 95% rename from src/pages/docs/observe/features/manual-tracing/add-events-exceptions-status.mdx rename to src/pages/docs/sdk/tracing/add-events-exceptions-status.mdx index c1ba95ba..2aa3e960 100644 --- a/src/pages/docs/observe/features/manual-tracing/add-events-exceptions-status.mdx +++ b/src/pages/docs/sdk/tracing/add-events-exceptions-status.mdx @@ -169,16 +169,16 @@ Spans capture timing and attributes, but they do not automatically record what h ## Next Steps - + Register a tracer provider and add instrumentation. - + Attach custom data to spans for filtering and evals. - + Access the active span or tracer at any point in your code. - + Use FITracer decorators and context managers for typed spans. diff --git a/src/pages/docs/observe/features/manual-tracing/advanced-tracing-examples.mdx b/src/pages/docs/sdk/tracing/advanced-tracing-examples.mdx similarity index 99% rename from src/pages/docs/observe/features/manual-tracing/advanced-tracing-examples.mdx rename to src/pages/docs/sdk/tracing/advanced-tracing-examples.mdx index d3ba6745..fd820977 100644 --- a/src/pages/docs/observe/features/manual-tracing/advanced-tracing-examples.mdx +++ b/src/pages/docs/sdk/tracing/advanced-tracing-examples.mdx @@ -675,16 +675,16 @@ Basic span creation works for synchronous, single-service code. But real applica ## Next Steps - + Register a tracer provider and add instrumentation. - + Use FITracer decorators and context managers for typed spans. - + Record exceptions and set span status for error visibility. - + Access and enrich the active span from anywhere in your code. diff --git a/src/pages/docs/observe/features/manual-tracing/annotating-using-api.mdx b/src/pages/docs/sdk/tracing/annotating-using-api.mdx similarity index 97% rename from src/pages/docs/observe/features/manual-tracing/annotating-using-api.mdx rename to src/pages/docs/sdk/tracing/annotating-using-api.mdx index b2d1d2b2..631da906 100644 --- a/src/pages/docs/observe/features/manual-tracing/annotating-using-api.mdx +++ b/src/pages/docs/sdk/tracing/annotating-using-api.mdx @@ -307,13 +307,13 @@ Each element in `result.errors` contains: ## Next Steps - + Register a tracer provider and add instrumentation. - + Attach custom data to spans for filtering and evals. - + Run evaluations directly inside a traced span. diff --git a/src/pages/docs/sdk/tracing/context-helpers.mdx b/src/pages/docs/sdk/tracing/context-helpers.mdx new file mode 100644 index 00000000..b4fbc33b --- /dev/null +++ b/src/pages/docs/sdk/tracing/context-helpers.mdx @@ -0,0 +1,213 @@ +--- +title: "Context helpers" +description: "Attach session, user, metadata, and tags to spans, and suppress tracing." +--- + + +Attach metadata, tags, session IDs, and user IDs to spans. These apply to all spans created within the scope. + + + + ```python + from fi_instrumentation import ( + using_session, using_user, using_metadata, + using_tags, using_prompt_template, using_attributes, + suppress_tracing + ) + + # Individual context managers + with using_session("session-abc-123"): + with using_user("user-456"): + response = client.chat.completions.create(...) + + with using_metadata({"environment": "production", "version": "2.1"}): + response = client.chat.completions.create(...) + + with using_tags(["rag-pipeline", "v2"]): + response = client.chat.completions.create(...) + + # Prompt template tracking + with using_prompt_template( + template="Answer {question} using {context}", + label="production", + version="v1.2", + variables={"question": "...", "context": "..."} + ): + response = client.chat.completions.create(...) + + # Combined - set everything at once + with using_attributes( + session_id="session-abc", + user_id="user-456", + metadata={"env": "prod"}, + tags=["rag", "v2"], + prompt_template="Answer {question}", + prompt_template_version="v1.2", + ): + response = client.chat.completions.create(...) + + # Suppress tracing for a block + with suppress_tracing(): + # These calls won't be traced + result = client.chat.completions.create(...) + ``` + + + ```typescript + import { + setSession, getSession, clearSession, + setUser, getUser, clearUser, + setMetadata, setTags, + setPromptTemplate, + getAttributesFromContext + } from "@traceai/fi-core"; + import { context } from "@opentelemetry/api"; + + // Build up context with multiple attributes + let ctx = context.active(); + ctx = setSession(ctx, { sessionId: "session-abc-123" }); + ctx = setUser(ctx, { userId: "user-456" }); + ctx = setMetadata(ctx, { environment: "production" }); + ctx = setTags(ctx, ["rag-pipeline", "v2"]); + ctx = setPromptTemplate(ctx, { + template: "Answer {{question}} using {{context}}", + variables: { question: "...", context: "..." }, + version: "v1.2", + }); + + // All spans created in this context inherit these attributes + context.with(ctx, async () => { + const response = await openai.chat.completions.create({ + model: "gpt-4o-mini", + messages: [{ role: "user", content: "Hello" }], + }); + }); + + // Read attributes back from context + const attrs = getAttributesFromContext(ctx); + ``` + + + Java uses `AutoCloseable` scopes with try-with-resources: + + ```java + import ai.traceai.ContextAttributes; + + // Session tracking + try (var ignored = ContextAttributes.usingSession("session-abc-123")) { + // All spans here get session.id and gen_ai.conversation.id + var response = tracedClient.createChatCompletion(params); + } + + // User tracking + try (var ignored = ContextAttributes.usingUser("user-456")) { + var response = tracedClient.createChatCompletion(params); + } + + // Metadata + try (var ignored = ContextAttributes.usingMetadata(Map.of( + "environment", "production", + "version", "2.1" + ))) { + var response = tracedClient.createChatCompletion(params); + } + + // Tags + try (var ignored = ContextAttributes.usingTags(List.of("rag-pipeline", "v2"))) { + var response = tracedClient.createChatCompletion(params); + } + + // Nest them for combined context + try (var s = ContextAttributes.usingSession("session-abc"); + var u = ContextAttributes.usingUser("user-456"); + var m = ContextAttributes.usingMetadata(Map.of("env", "prod"))) { + var response = tracedClient.createChatCompletion(params); + } + + // Read current attributes + Map attrs = ContextAttributes.getAttributesFromContext(); + ``` + + + C# uses `IDisposable` scopes with `using` statements: + + ```csharp + using FIInstrumentation.Context; + + // Session and user tracking + using (ContextAttributes.UsingSession("session-abc-123")) + using (ContextAttributes.UsingUser("user-456")) + { + tracer.Llm("llm-call", span => + { + // span automatically gets session.id and user.id + span.SetInput("Hello!"); + }); + } + + // Metadata and tags + using (ContextAttributes.UsingMetadata(new Dictionary + { + ["environment"] = "production", + ["version"] = "2.1" + })) + using (ContextAttributes.UsingTags(new List { "rag-pipeline", "v2" })) + { + tracer.Chain("pipeline", span => { /* ... */ }); + } + + // Prompt template tracking + using (ContextAttributes.UsingPromptTemplate( + template: "Answer {question} using {context}", + label: "production", + version: "v1.2", + variables: new Dictionary + { + ["question"] = "...", + ["context"] = "..." + } + )) + { + tracer.Llm("templated-call", span => { /* ... */ }); + } + + // Combined - set everything at once + using (ContextAttributes.UsingAttributes( + sessionId: "session-abc", + userId: "user-456", + metadata: new Dictionary { ["env"] = "prod" }, + tags: new List { "rag", "v2" } + )) + { + tracer.Chain("full-context", span => { /* ... */ }); + } + ``` + + + +## Suppress Tracing + +Temporarily disable tracing for a block of code. Useful for health checks, internal calls, or operations you don't want in your traces. Available in Python and C# only - Java and TypeScript don't have this API. + + + + ```python + from fi_instrumentation import suppress_tracing + + with suppress_tracing(): + # Nothing in this block is traced + result = client.chat.completions.create(...) + ``` + + + ```csharp + using FIInstrumentation.Context; + + using (new SuppressTracing()) + { + // Nothing in this block is traced + } + ``` + + + diff --git a/src/pages/docs/observe/features/manual-tracing/create-tool-spans.mdx b/src/pages/docs/sdk/tracing/create-tool-spans.mdx similarity index 97% rename from src/pages/docs/observe/features/manual-tracing/create-tool-spans.mdx rename to src/pages/docs/sdk/tracing/create-tool-spans.mdx index c78dd39f..ac4f9760 100644 --- a/src/pages/docs/observe/features/manual-tracing/create-tool-spans.mdx +++ b/src/pages/docs/sdk/tracing/create-tool-spans.mdx @@ -241,16 +241,16 @@ LLM agents often call external tools (APIs, databases, code interpreters), but t ## Next Steps - + Use FITracer decorators and context managers for typed spans. - + Attach custom data to spans for filtering and evals. - + Record exceptions and set span status for error visibility. - + Register a tracer provider and add instrumentation. diff --git a/src/pages/docs/sdk/tracing/environment-variables.mdx b/src/pages/docs/sdk/tracing/environment-variables.mdx new file mode 100644 index 00000000..76cdb75a --- /dev/null +++ b/src/pages/docs/sdk/tracing/environment-variables.mdx @@ -0,0 +1,28 @@ +--- +title: "Environment variables" +description: "Environment variables that configure traceAI tracing." +--- + + +All languages read from the same set of environment variables: + +| Variable | Purpose | Default | +|----------|---------|---------| +| `FI_API_KEY` | Authentication | required | +| `FI_SECRET_KEY` | Authentication | required | +| `FI_BASE_URL` | HTTP collector endpoint | `https://api.futureagi.com` | +| `FI_GRPC_URL` | gRPC collector endpoint | `https://grpc.futureagi.com` | +| `FI_PROJECT_NAME` | Default project name | None | +| `FI_PROJECT_VERSION_NAME` | Default version | None | +| `FI_HIDE_INPUTS` | Redact inputs | False | +| `FI_HIDE_OUTPUTS` | Redact outputs | False | +| `FI_HIDE_INPUT_MESSAGES` | Redact input messages | False | +| `FI_HIDE_OUTPUT_MESSAGES` | Redact output messages | False | +| `FI_HIDE_INPUT_IMAGES` | Redact input images | False | +| `FI_HIDE_INPUT_TEXT` | Redact input text | False | +| `FI_HIDE_OUTPUT_TEXT` | Redact output text | False | +| `FI_HIDE_EMBEDDING_VECTORS` | Redact embedding vectors | False | +| `FI_HIDE_LLM_INVOCATION_PARAMETERS` | Redact model parameters | False | +| `FI_BASE64_IMAGE_MAX_LENGTH` | Max base64 image chars | 32000 | +| `FI_PII_REDACTION` | Auto-mask PII (Python) | False | + diff --git a/src/pages/docs/sdk/tracing/eval-tags.mdx b/src/pages/docs/sdk/tracing/eval-tags.mdx new file mode 100644 index 00000000..8f312731 --- /dev/null +++ b/src/pages/docs/sdk/tracing/eval-tags.mdx @@ -0,0 +1,125 @@ +--- +title: "EvalTags" +description: "Attach evaluations to traces with EvalTags." +--- + + +EvalTags let you configure automatic evaluations that run server-side on your traced spans. Attach them during `register()` and the platform scores spans as they arrive. + + + + ```python + from fi_instrumentation import register + from fi_instrumentation.fi_types import ( + ProjectType, EvalTag, EvalTagType, + EvalSpanKind, EvalName, ModelChoices + ) + + trace_provider = register( + project_name="my-project", + project_type=ProjectType.EXPERIMENT, + project_version_name="v1.0", + eval_tags=[ + EvalTag( + type=EvalTagType.OBSERVATION_SPAN, + value=EvalSpanKind.LLM, + eval_name=EvalName.GROUNDEDNESS, + model=ModelChoices.TURING_FLASH, + ), + EvalTag( + type=EvalTagType.OBSERVATION_SPAN, + value=EvalSpanKind.LLM, + eval_name=EvalName.TOXICITY, + model=ModelChoices.TURING_FLASH, + ), + ], + ) + ``` + + + ```typescript + import { + register, ProjectType, EvalTag, + EvalTagType, EvalSpanKind, EvalName, ModelChoices + } from "@traceai/fi-core"; + + const tracerProvider = register({ + projectName: "my-project", + projectType: ProjectType.EXPERIMENT, + projectVersionName: "v1.0", + evalTags: [ + await EvalTag.create({ + type: EvalTagType.OBSERVATION_SPAN, + value: EvalSpanKind.LLM, + eval_name: EvalName.GROUNDEDNESS, + model: ModelChoices.TURING_FLASH, + }), + await EvalTag.create({ + type: EvalTagType.OBSERVATION_SPAN, + value: EvalSpanKind.LLM, + eval_name: EvalName.TOXICITY, + model: ModelChoices.TURING_FLASH, + }), + ], + }); + ``` + + + `EvalTag.create()` is async in TypeScript because it validates the eval configuration with the server. + + + + ```csharp + using FIInstrumentation; + using FIInstrumentation.Types; + + var tracer = TraceAI.Register(opts => + { + opts.ProjectName = "my-project"; + opts.ProjectType = ProjectType.Experiment; + opts.ProjectVersionName = "v1.0"; + opts.EvalTags = new List + { + new EvalTag(EvalSpanKind.Llm, EvalName.Groundedness) + { + Model = ModelChoices.TuringFlash, + }, + new EvalTag(EvalSpanKind.Llm, EvalName.Toxicity) + { + Model = ModelChoices.TuringFlash, + }, + }; + }); + ``` + + + +## EvalSpanKind + +Which span types to evaluate: + +| Value | Description | +|-------|-------------| +| `LLM` | Language model calls | +| `RETRIEVER` | Document retrieval spans | +| `TOOL` | Tool/function calls | +| `AGENT` | Agent spans | +| `EMBEDDING` | Embedding generation | +| `RERANKER` | Re-ranking operations | + +## ModelChoices + +Which evaluation model to use: + +| Value | Description | +|-------|-------------| +| `TURING_FLASH` | Fast evaluation model | +| `TURING_SMALL` | Small evaluation model | +| `TURING_LARGE` | High-accuracy evaluation model | +| `PROTECT` | Safety-focused model | +| `PROTECT_FLASH` | Fast safety model | + + + EvalTags only work with `ProjectType.EXPERIMENT`. For production monitoring without evals, use `ProjectType.OBSERVE`. + + diff --git a/src/pages/docs/sdk/tracing/fitracer.mdx b/src/pages/docs/sdk/tracing/fitracer.mdx new file mode 100644 index 00000000..c4fc9c5d --- /dev/null +++ b/src/pages/docs/sdk/tracing/fitracer.mdx @@ -0,0 +1,272 @@ +--- +title: "FITracer & custom spans" +description: "Create custom spans, set span kinds, and use FITracer decorators." +--- + + +Beyond auto-instrumentation, `FITracer` lets you create custom spans for your own logic - agent steps, chain stages, tool calls, or any operation you want to trace. + +## Span Kinds + +All languages share the same span kinds: + +| Kind | Use for | +|------|---------| +| `LLM` | Language model inference calls | +| `CHAIN` | Sequential pipeline steps | +| `AGENT` | Autonomous agent actions | +| `TOOL` | Tool/function calls | +| `EMBEDDING` | Vector generation | +| `RETRIEVER` | Document retrieval (RAG) | +| `RERANKER` | Re-ranking operations | +| `GUARDRAIL` | Safety/validation checks | +| `EVALUATOR` | Quality scoring | +| `UNKNOWN` | Unspecified or unexpected span type | +| `WORKFLOW` | Custom pipeline steps (Java only) | +| `CONVERSATION` | Voice/conversational AI (Java/C#) | +| `VECTOR_DB` | Vector database operations (Java/C#) | + +## Decorators and Convenience Methods + + + + Python's `FITracer` provides decorators for clean span creation: + + ```python + from fi_instrumentation import register + from fi_instrumentation.fi_types import ProjectType + + trace_provider = register( + project_name="my-project", + project_type=ProjectType.OBSERVE, + ) + tracer = trace_provider.get_tracer(__name__) + + # Use the FITracer wrapper for decorators + from fi_instrumentation import FITracer + fi_tracer = FITracer(tracer) + + @fi_tracer.agent(name="research-agent") + def research_agent(query): + # This entire function becomes an AGENT span + results = search(query) + return summarize(results) + + @fi_tracer.chain(name="rag-pipeline") + def rag_pipeline(question): + docs = retrieve(question) + return generate(question, docs) + + @fi_tracer.tool( + name="web-search", + description="Searches the web", + parameters={"query": {"type": "string"}} + ) + def web_search(query): + return requests.get(f"https://api.search.com?q={query}").json() + ``` + + You can also use context managers for manual span creation: + + ```python + from fi_instrumentation.fi_types import FiSpanKindValues + + with fi_tracer.start_as_current_span( + "llm-call", + fi_span_kind=FiSpanKindValues.LLM, + ) as span: + span.set_input(value="What is Python?") + response = call_llm("What is Python?") + span.set_output(value=response) + span.set_attributes({ + "gen_ai.request.model": "gpt-4o", + "gen_ai.usage.input_tokens": 10, + "gen_ai.usage.output_tokens": 150, + }) + ``` + + + TypeScript uses OpenTelemetry's standard `startActiveSpan` pattern: + + ```typescript + import { trace } from "@opentelemetry/api"; + + const tracer = trace.getTracer("my-app"); + + // Manual span creation + tracer.startActiveSpan("rag-pipeline", (span) => { + span.setAttribute("gen_ai.span.kind", "CHAIN"); + span.setAttribute("input.value", question); + + const docs = retrieve(question); + const result = generate(question, docs); + + span.setAttribute("output.value", result); + span.end(); + return result; + }); + ``` + + Context management functions let you set session, user, and metadata: + + ```typescript + import { + setSession, setUser, setMetadata, setTags, + getAttributesFromContext + } from "@traceai/fi-core"; + import { context } from "@opentelemetry/api"; + + const ctx = setSession(context.active(), { sessionId: "sess-123" }); + const ctx2 = setUser(ctx, { userId: "user-456" }); + + context.with(ctx2, () => { + // All spans created here inherit session and user + tracer.startActiveSpan("operation", (span) => { + // span automatically gets session.id and user.id + span.end(); + }); + }); + ``` + + + Java offers both lambda-based and manual span creation: + + ```java + import ai.traceai.FITracer; + import ai.traceai.FISpanKind; + + FITracer tracer = TraceAI.getTracer(); + + // Lambda-based - auto-manages span lifecycle + String result = tracer.trace("rag-pipeline", FISpanKind.CHAIN, (span) -> { + tracer.setInputValue(span, question); + + String docs = tracer.trace("retrieve", FISpanKind.RETRIEVER, (rSpan) -> { + tracer.setInputValue(rSpan, question); + var retrieved = vectorDb.search(question); + tracer.setOutputValue(rSpan, tracer.toJson(retrieved)); + return retrieved; + }); + + String answer = tracer.trace("generate", FISpanKind.LLM, (lSpan) -> { + tracer.setInputMessages(lSpan, List.of( + tracer.message("system", "Answer using the context."), + tracer.message("user", question) + )); + var resp = llm.generate(question, docs); + tracer.setOutputMessages(lSpan, List.of( + tracer.message("assistant", resp) + )); + tracer.setTokenCounts(lSpan, 50, 200, 250); + return resp; + }); + + tracer.setOutputValue(span, answer); + return answer; + }); + ``` + + Manual span creation for more control: + + ```java + import io.opentelemetry.api.trace.Span; + import io.opentelemetry.context.Context; + + Span span = tracer.startSpan("tool-call", FISpanKind.TOOL); + try { + tracer.setInputValue(span, inputJson); + String result = executeTool(inputJson); + tracer.setOutputValue(span, result); + span.setStatus(StatusCode.OK); + } catch (Exception e) { + tracer.setError(span, e); + } finally { + span.end(); + } + ``` + + + C# provides typed convenience methods for each span kind: + + ```csharp + var tracer = TraceAI.Register(opts => + { + opts.ProjectName = "my-project"; + opts.ProjectType = ProjectType.Observe; + }); + + // Convenience methods for each span kind + var result = tracer.Chain("rag-pipeline", span => + { + span.SetInput("What is quantum computing?"); + + var docs = tracer.Tool("vector-search", toolSpan => + { + toolSpan.SetTool("search", "Searches vector DB"); + toolSpan.SetInput("quantum computing"); + var results = vectorDb.Search("quantum computing"); + toolSpan.SetOutput(results); + return results; + }); + + var answer = tracer.Llm("generate", llmSpan => + { + llmSpan.SetAttribute(SemanticConventions.GenAiRequestModel, "gpt-4o"); + llmSpan.SetInputMessages(new List> + { + FITracer.Message("user", "What is quantum computing?") + }); + var resp = llm.Generate("What is quantum computing?", docs); + llmSpan.SetOutputMessages(new List> + { + FITracer.Message("assistant", resp) + }); + llmSpan.SetTokenCounts(50, 200, 250); + return resp; + }); + + span.SetOutput(answer); + return answer; + }); + + // Async variants + await tracer.AgentAsync("research-agent", async span => + { + span.SetInput("Research topic X"); + var result = await RunResearchAsync("topic X"); + span.SetOutput(result); + }); + ``` + + Manual span creation: + + ```csharp + using var span = tracer.StartSpan("custom-op", FISpanKind.Chain); + span.SetInput("input data"); + span.SetOutput("output data"); + // span.Dispose() ends the span automatically + ``` + + + +## FISpan Methods + +All languages provide methods on the span object for setting structured data: + +| Method | Description | Available in | +|--------|-------------|-------------| +| `set_input(value, mime_type=)` / `SetInput(value, mimeType)` | Set span input value (text or JSON). `mime_type` accepts `"text/plain"` or `"application/json"` | Python, C# | +| `set_output(value, mime_type=)` / `SetOutput(value, mimeType)` | Set span output value | Python, C# | +| `set_tool(name, description, parameters)` / `SetTool(...)` | Attach tool metadata | Python, C# | +| `set_attributes(dict)` / `SetAttribute(key, value)` | Set custom attributes | All | +| `setInputValue(span, value)` | Set input on span | Java | +| `setOutputValue(span, value)` | Set output on span | Java | +| `setInputMessages(span, messages)` / `SetInputMessages(messages)` | Set chat message history | Java, C# | +| `setOutputMessages(span, messages)` / `SetOutputMessages(messages)` | Set response messages | Java, C# | +| `setTokenCounts(span, in, out, total)` / `SetTokenCounts(in, out, total)` | Set token usage | Java, C# | +| `setError(span, exception)` / `SetError(exception)` | Record an exception | Java, C# | + + + In Java, these methods live on `FITracer` and take the span as the first argument (e.g. `tracer.setInputValue(span, value)`). In Python and C#, they're called directly on the span object. + + diff --git a/src/pages/docs/observe/features/manual-tracing/get-current-span-context.mdx b/src/pages/docs/sdk/tracing/get-current-span-context.mdx similarity index 95% rename from src/pages/docs/observe/features/manual-tracing/get-current-span-context.mdx rename to src/pages/docs/sdk/tracing/get-current-span-context.mdx index d4157bcf..64636ff0 100644 --- a/src/pages/docs/observe/features/manual-tracing/get-current-span-context.mdx +++ b/src/pages/docs/sdk/tracing/get-current-span-context.mdx @@ -133,16 +133,16 @@ Spans and tracers are usually created at the top of a request, but the functions ## Next Steps - + Register a tracer provider and add instrumentation. - + Use FITracer decorators and context managers for typed spans. - + Attach custom data to spans for filtering and evals. - + Group traces into sessions and link them to end users. diff --git a/src/pages/docs/observe/features/manual-tracing/in-line-evals.mdx b/src/pages/docs/sdk/tracing/in-line-evals.mdx similarity index 94% rename from src/pages/docs/observe/features/manual-tracing/in-line-evals.mdx rename to src/pages/docs/sdk/tracing/in-line-evals.mdx index 2aa5b933..0b2285f9 100644 --- a/src/pages/docs/observe/features/manual-tracing/in-line-evals.mdx +++ b/src/pages/docs/sdk/tracing/in-line-evals.mdx @@ -98,13 +98,13 @@ Evaluation results are most useful when they sit next to the data that produced ## Next Steps - + Register a tracer provider and add instrumentation. - + Use FITracer decorators and context managers for typed spans. - + Attach custom data to spans for filtering and evals. diff --git a/src/pages/docs/sdk/tracing/index.mdx b/src/pages/docs/sdk/tracing/index.mdx new file mode 100644 index 00000000..6fabff28 --- /dev/null +++ b/src/pages/docs/sdk/tracing/index.mdx @@ -0,0 +1,158 @@ +--- +title: "Tracing: OpenTelemetry SDK for Future AGI AI Apps" +description: "Set up OpenTelemetry tracing across Python, TypeScript, Java, and C#. Auto-instrument 45+ frameworks or create custom spans with FITracer." +--- + + +- `register()` sets up the tracer provider in two lines, all languages +- Auto-instrument with `traceai-*` packages (45+ frameworks) or create custom spans with `FITracer` +- Context helpers attach session, user, metadata, and tags to all spans in a block +- TraceConfig controls privacy masking, PII redaction covers 6 data types automatically + + +The pattern is the same across all four languages: call `register()` once to set up the provider, then either auto-instrument your frameworks or use `FITracer` for custom spans. LLM calls, retrieval steps, and agent actions get captured as OpenTelemetry spans and sent to your dashboard. + + + Requires `FI_API_KEY` and `FI_SECRET_KEY` in your environment. For conceptual background on traces, spans, and attributes, see the [Tracing guide](/docs/tracing/auto). + + +## Quick Example + + + + ```bash + pip install fi-instrumentation-otel traceai-openai + ``` + + ```python + from fi_instrumentation import register + from fi_instrumentation.fi_types import ProjectType + from traceai_openai import OpenAIInstrumentor + + # 1. Register the tracer provider + trace_provider = register( + project_name="my-project", + project_type=ProjectType.OBSERVE, + ) + + # 2. Instrument your framework + OpenAIInstrumentor().instrument(tracer_provider=trace_provider) + + # 3. Use OpenAI as normal - all calls are now traced + import openai + client = openai.OpenAI() + response = client.chat.completions.create( + model="gpt-4o-mini", + messages=[{"role": "user", "content": "What is Python?"}], + ) + ``` + + + ```bash + npm install @traceai/openai @traceai/fi-core @opentelemetry/instrumentation + ``` + + ```typescript + import { register, ProjectType } from "@traceai/fi-core"; + import { OpenAIInstrumentation } from "@traceai/openai"; + import { registerInstrumentations } from "@opentelemetry/instrumentation"; + import OpenAI from "openai"; + + const tracerProvider = register({ + projectName: "my-project", + projectType: ProjectType.OBSERVE, + }); + + registerInstrumentations({ + tracerProvider, + instrumentations: [new OpenAIInstrumentation()], + }); + + const openai = new OpenAI(); + const response = await openai.chat.completions.create({ + model: "gpt-4o-mini", + messages: [{ role: "user", content: "Hello!" }], + }); + ``` + + + ```xml + + + com.github.future-agi.traceAI + traceai-spring-boot-starter + v1.0.0 + + + com.github.future-agi.traceAI + traceai-java-openai + v1.0.0 + + ``` + + ```java + import ai.traceai.TraceAI; + import ai.traceai.TraceConfig; + import ai.traceai.openai.TracedOpenAIClient; + + // Initialize from environment variables + TraceAI.initFromEnvironment(); + + // Wrap your client + TracedOpenAIClient tracedClient = new TracedOpenAIClient(openAIClient); + var response = tracedClient.createChatCompletion(params); + ``` + + Set `FI_API_KEY`, `FI_SECRET_KEY`, `FI_BASE_URL`, and `FI_PROJECT_NAME` as environment variables. + + + ```bash + dotnet add package fi-instrumentation-otel + ``` + + ```csharp + using FIInstrumentation; + using FIInstrumentation.Types; + + var tracer = TraceAI.Register(opts => + { + opts.ProjectName = "my-project"; + opts.ProjectType = ProjectType.Observe; + }); + + // Create traced LLM calls with convenience methods + var result = tracer.Llm("openai-call", span => + { + span.SetInput("What is C#?"); + var response = CallOpenAI("What is C#?"); + span.SetOutput(response); + return response; + }); + + TraceAI.Shutdown(); + ``` + + + +## Related + + + + Concepts, manual tracing, and per-framework setup guides. + + + Setup guides for all 45+ supported frameworks. + + + Score traced outputs with 76+ metrics. + + + Store test data and run batch evaluations. + + + Guard inputs and outputs with safety rules. + + + Test voice AI agents with simulated personas. + + diff --git a/src/pages/docs/observe/features/manual-tracing/instrument-with-traceai-helpers.mdx b/src/pages/docs/sdk/tracing/instrument-with-traceai-helpers.mdx similarity index 97% rename from src/pages/docs/observe/features/manual-tracing/instrument-with-traceai-helpers.mdx rename to src/pages/docs/sdk/tracing/instrument-with-traceai-helpers.mdx index 34d4368e..30bfdff6 100644 --- a/src/pages/docs/observe/features/manual-tracing/instrument-with-traceai-helpers.mdx +++ b/src/pages/docs/sdk/tracing/instrument-with-traceai-helpers.mdx @@ -323,22 +323,22 @@ Manual tracing with raw OpenTelemetry means writing a lot of setup code for ever ## Next Steps - + Register a tracer provider and add instrumentation. - + Attach custom data to spans for filtering and evals. - + Group traces into sessions and link them to end users. - + Redact sensitive data with TraceConfig before export. Browse all supported framework instrumentors. - + Register an Observe project and start capturing traces. diff --git a/src/pages/docs/sdk/tracing/instrumentors.mdx b/src/pages/docs/sdk/tracing/instrumentors.mdx new file mode 100644 index 00000000..db2cb62e --- /dev/null +++ b/src/pages/docs/sdk/tracing/instrumentors.mdx @@ -0,0 +1,183 @@ +--- +title: "Instrumentors" +description: "Auto-instrument frameworks, clean up, and instrument other languages." +--- + + +Each framework has its own instrumentor package. Install the one for your framework and call `.instrument()`. + +```python +# Pattern is the same for every framework: +from traceai_ import Instrumentor +Instrumentor().instrument(tracer_provider=trace_provider) +``` + + + + | Package | Framework | Instrumentor class | + |---------|-----------|-------------------| + | `traceai-openai` | OpenAI | `OpenAIInstrumentor` | + | `traceai-anthropic` | Anthropic | `AnthropicInstrumentor` | + | `traceai-google-genai` | Google GenAI | `GoogleGenAIInstrumentor` | + | `traceai-vertexai` | Vertex AI | `VertexAIInstrumentor` | + | `traceai-bedrock` | AWS Bedrock | `BedrockInstrumentor` | + | `traceai-mistralai` | Mistral AI | `MistralAIInstrumentor` | + | `traceai-groq` | Groq | `GroqInstrumentor` | + | `traceai-litellm` | LiteLLM | `LiteLLMInstrumentor` | + | `traceai-cohere` | Cohere | `CohereInstrumentor` | + | `traceai-ollama` | Ollama | `OllamaInstrumentor` | + | `traceai-deepseek` | DeepSeek | `DeepSeekInstrumentor` | + | `traceai-together` | Together AI | `TogetherInstrumentor` | + | `traceai-fireworks` | Fireworks AI | `FireworksInstrumentor` | + | `traceai-cerebras` | Cerebras | `CerebrasInstrumentor` | + | `traceai-xai` | xAI / Grok | `XAIInstrumentor` | + | `traceai-vllm` | vLLM | `VLLMInstrumentor` | + | `traceai-portkey` | Portkey | `PortkeyInstrumentor` | + | `traceai-huggingface` | HuggingFace | `HuggingFaceInstrumentor` | + + + | Package | Framework | Instrumentor class | + |---------|-----------|-------------------| + | `traceai-langchain` | LangChain / LangGraph | `LangChainInstrumentor` | + | `traceai-llamaindex` | LlamaIndex | `LlamaIndexInstrumentor` | + | `traceai-crewai` | CrewAI | `CrewAIInstrumentor` | + | `traceai-openai-agents` | OpenAI Agents SDK | `OpenAIAgentsInstrumentor` | + | `traceai-autogen` | Microsoft AutoGen | `AutoGenInstrumentor` | + | `traceai-smolagents` | HuggingFace SmolAgents | `SmolAgentsInstrumentor` | + | `traceai-google-adk` | Google Agent Dev Kit | `GoogleADKInstrumentor` | + | `traceai-claude-agent-sdk` | Claude Agent SDK | `ClaudeAgentSDKInstrumentor` | + | `traceai-pydantic-ai` | Pydantic AI | `PydanticAIInstrumentor` | + | `traceai-strands` | AWS Strands Agents | `StrandsInstrumentor` | + | `traceai-agno` | Agno | `AgnoInstrumentor` | + | `traceai-beeai` | IBM BeeAI | `BeeAIInstrumentor` | + | `traceai-haystack` | Haystack | `HaystackInstrumentor` | + | `traceai-dspy` | DSPy | `DSPyInstrumentor` | + | `traceai-guardrails` | Guardrails AI | `GuardrailsInstrumentor` | + | `traceai-instructor` | Instructor | `InstructorInstrumentor` | + | `traceai-mcp` | Model Context Protocol | `MCPInstrumentor` | + + + | Package | Framework | Instrumentor class | + |---------|-----------|-------------------| + | `traceai-pipecat` | Pipecat | `PipecatInstrumentor` | + | `traceai-livekit` | LiveKit | `LiveKitInstrumentor` | + + + | Package | Framework | Instrumentor class | + |---------|-----------|-------------------| + | `traceai-pinecone` | Pinecone | `PineconeInstrumentor` | + | `traceai-chromadb` | ChromaDB | `ChromaDBInstrumentor` | + | `traceai-qdrant` | Qdrant | `QdrantInstrumentor` | + | `traceai-weaviate` | Weaviate | `WeaviateInstrumentor` | + | `traceai-milvus` | Milvus | `MilvusInstrumentor` | + | `traceai-lancedb` | LanceDB | `LanceDBInstrumentor` | + | `traceai-mongodb` | MongoDB | `MongoDBInstrumentor` | + | `traceai-pgvector` | pgvector | `PgVectorInstrumentor` | + | `traceai-redis` | Redis | `RedisInstrumentor` | + + + +## Cleanup + +To remove instrumentation (useful in tests or serverless cleanup): + + + + ```python + OpenAIInstrumentor().uninstrument() + ``` + + + ```java + TraceAI.shutdown(); // Flushes remaining spans and shuts down + ``` + + + ```csharp + TraceAI.Shutdown(); // Flushes remaining spans and shuts down + ``` + + + +For per-framework setup guides with full examples, see the [Auto-Instrumentation docs](/docs/tracing/auto). + +## Other Languages + +The tables above show Python packages. TypeScript, Java, and C# have their own instrumentation libraries: + + + + TypeScript packages follow the `@traceai/` pattern. All use OpenTelemetry's `registerInstrumentations()`. + + ```typescript + import { registerInstrumentations } from "@opentelemetry/instrumentation"; + import { OpenAIInstrumentation } from "@traceai/openai"; + import { AnthropicInstrumentation } from "@traceai/anthropic"; + import { LangChainInstrumentation } from "@traceai/langchain"; + import { PineconeInstrumentation } from "@traceai/pinecone"; + + registerInstrumentations({ + tracerProvider, + instrumentations: [ + new OpenAIInstrumentation(), + new AnthropicInstrumentation(), + new LangChainInstrumentation(), + new PineconeInstrumentation(), + ], + }); + ``` + + 40+ packages available including all LLM providers, frameworks, and vector DBs from the Python list, plus `@traceai/vercel` for Vercel/Next.js and `@traceai/mastra`. + + + Java uses the `Traced*` wrapper pattern. Each integration wraps the native client: + + ```java + // LLM Providers + TracedOpenAIClient traced = new TracedOpenAIClient(openAIClient); + TracedAnthropicClient traced = new TracedAnthropicClient(anthropicClient); + TracedBedrockRuntimeClient traced = new TracedBedrockRuntimeClient(bedrockClient); + TracedGenerativeModel traced = new TracedGenerativeModel(model); // Google GenAI + TracedOllamaAPI traced = new TracedOllamaAPI(ollamaAPI); + TracedCohereClient traced = new TracedCohereClient(cohereClient); + TracedWatsonxAI traced = new TracedWatsonxAI(watsonxClient); + + // Vector Databases + TracedPineconeIndex traced = new TracedPineconeIndex(index, "my-index"); + TracedQdrantClient traced = new TracedQdrantClient(qdrantClient); + TracedMilvusClient traced = new TracedMilvusClient(milvusClient); + TracedChromaCollection traced = new TracedChromaCollection(collection); + TracedMongoVectorSearch traced = new TracedMongoVectorSearch(collection); + TracedRedisVectorSearch traced = new TracedRedisVectorSearch(jedis); + TracedSearchClient traced = new TracedSearchClient(searchClient); // Azure Search + TracedPgVectorStore traced = new TracedPgVectorStore(connection); + TracedElasticsearchClient traced = new TracedElasticsearchClient(esClient); + + // Framework integrations + TracedChatLanguageModel traced = new TracedChatLanguageModel(model, tracer, "openai"); // LangChain4j + TracedChatModel traced = new TracedChatModel(chatModel, tracer, "openai"); // Spring AI + TracedKernel traced = new TracedKernel(kernel, tracer); // Semantic Kernel + ``` + + Maven coordinates: `com.github.future-agi.traceAI:traceai-java-:v1.0.0` + + + C# uses manual tracing via `FITracer`. No auto-instrumentation wrappers yet - use the convenience methods (`Llm()`, `Chain()`, `Agent()`, `Tool()`) to create spans around your calls. + + ```csharp + // Wrap any LLM call + var response = tracer.Llm("openai-call", span => + { + span.SetAttribute(SemanticConventions.GenAiRequestModel, "gpt-4o"); + span.SetInput(prompt); + var result = CallOpenAI(prompt); + span.SetOutput(result); + span.SetTokenCounts(inputTokens, outputTokens, totalTokens); + return result; + }); + ``` + + Install: `dotnet add package fi-instrumentation-otel` + + + diff --git a/src/pages/docs/observe/features/manual-tracing/langfuse-integration.mdx b/src/pages/docs/sdk/tracing/langfuse-integration.mdx similarity index 95% rename from src/pages/docs/observe/features/manual-tracing/langfuse-integration.mdx rename to src/pages/docs/sdk/tracing/langfuse-integration.mdx index 331acd61..e961e7e8 100644 --- a/src/pages/docs/observe/features/manual-tracing/langfuse-integration.mdx +++ b/src/pages/docs/sdk/tracing/langfuse-integration.mdx @@ -123,10 +123,10 @@ Langfuse provides tracing but does not have a built-in evaluation engine. This i Learn how to run evaluations using the Future AGI AI Evaluations library. - + Run evaluations directly inside a traced span with Future AGI tracing. - + Register a tracer provider and add instrumentation. diff --git a/src/pages/docs/observe/features/manual-tracing/log-prompt-templates.mdx b/src/pages/docs/sdk/tracing/log-prompt-templates.mdx similarity index 94% rename from src/pages/docs/observe/features/manual-tracing/log-prompt-templates.mdx rename to src/pages/docs/sdk/tracing/log-prompt-templates.mdx index c2588f61..945e2782 100644 --- a/src/pages/docs/observe/features/manual-tracing/log-prompt-templates.mdx +++ b/src/pages/docs/sdk/tracing/log-prompt-templates.mdx @@ -112,16 +112,16 @@ LLM outputs depend entirely on the prompt, but the prompt itself is not captured ## Next Steps - + Register a tracer provider and add instrumentation. - + Attach custom data to spans for filtering and evals. - + Use FITracer decorators and context managers for typed spans. - + Group traces into sessions and link them to end users. diff --git a/src/pages/docs/observe/features/manual-tracing/mask-span-attributes.mdx b/src/pages/docs/sdk/tracing/mask-span-attributes.mdx similarity index 95% rename from src/pages/docs/observe/features/manual-tracing/mask-span-attributes.mdx rename to src/pages/docs/sdk/tracing/mask-span-attributes.mdx index 3865a1ca..4cf902fc 100644 --- a/src/pages/docs/observe/features/manual-tracing/mask-span-attributes.mdx +++ b/src/pages/docs/sdk/tracing/mask-span-attributes.mdx @@ -101,13 +101,13 @@ Traces often contain sensitive data: user messages, API responses, PII, or large ## Next Steps - + Register a tracer provider and add instrumentation. - + Attach custom data to spans for filtering and evals. - + Use FITracer decorators and context managers for typed spans. diff --git a/src/pages/docs/sdk/tracing/register.mdx b/src/pages/docs/sdk/tracing/register.mdx new file mode 100644 index 00000000..83908feb --- /dev/null +++ b/src/pages/docs/sdk/tracing/register.mdx @@ -0,0 +1,182 @@ +--- +title: "register()" +description: "Set up the tracer provider and connect to Future AGI with register()." +--- + + +Creates an OpenTelemetry tracer provider configured to export spans to your Future AGI dashboard. + + + + ```python + from fi_instrumentation import register + from fi_instrumentation.fi_types import ProjectType, Transport + + trace_provider = register( + project_name="my-project", + project_type=ProjectType.OBSERVE, + transport=Transport.HTTP, + batch=True, + verbose=True, + ) + ``` + + | Parameter | Type | Default | Description | + |-----------|------|---------|-------------| + | `project_name` | str / None | `FI_PROJECT_NAME` env var | Project identifier in the dashboard | + | `project_type` | ProjectType | `EXPERIMENT` | `EXPERIMENT` (dev, supports eval tags) or `OBSERVE` (production) | + | `project_version_name` | str / None | None | Version label (EXPERIMENT only) | + | `eval_tags` | list / None | None | Evaluation configs for automated span scoring (EXPERIMENT only) | + | `metadata` | dict / None | None | Custom metadata attached to all spans | + | `batch` | bool | True | True = BatchSpanProcessor, False = SimpleSpanProcessor | + | `set_global_tracer_provider` | bool | False | Register as the global OpenTelemetry default | + | `headers` | dict / None | None | Custom HTTP headers (auto-populated from API keys if not set) | + | `verbose` | bool | True | Print configuration details on startup | + | `transport` | Transport | `HTTP` | `HTTP` or `GRPC` | + | `semantic_convention` | SemanticConvention | `FI` | Attribute naming convention | + + **Returns:** `TracerProvider` - pass this to `.instrument(tracer_provider=...)` on any instrumentor. + + + ```typescript + import { register, ProjectType, Transport } from "@traceai/fi-core"; + + const tracerProvider = register({ + projectName: "my-project", + projectType: ProjectType.OBSERVE, + transport: Transport.HTTP, + batch: true, + verbose: true, + }); + ``` + + | Parameter | Type | Default | Description | + |-----------|------|---------|-------------| + | `projectName` | string | `FI_PROJECT_NAME` env var | Project identifier | + | `projectType` | ProjectType | `EXPERIMENT` | `EXPERIMENT` or `OBSERVE` | + | `projectVersionName` | string | undefined | Version label (EXPERIMENT only) | + | `evalTags` | EvalTag[] | undefined | Evaluation configs (EXPERIMENT only) | + | `sessionName` | string | undefined | Session name (OBSERVE only) | + | `metadata` | Record | undefined | Custom metadata | + | `batch` | boolean | false | Use batch span processor | + | `setGlobalTracerProvider` | boolean | true | Register as global provider | + | `headers` | FIHeaders | undefined | Custom HTTP headers | + | `verbose` | boolean | false | Verbose logging | + | `endpoint` | string | `FI_BASE_URL` | Custom endpoint | + | `transport` | Transport | `HTTP` | `HTTP` or `GRPC` | + + **Returns:** `FITracerProvider` + + + ```java + import ai.traceai.TraceAI; + import ai.traceai.TraceConfig; + + // Option 1: From environment variables + TraceAI.initFromEnvironment(); + + // Option 2: Programmatic configuration + TraceAI.init(TraceConfig.builder() + .baseUrl("https://api.futureagi.com") + .apiKey("your-api-key") + .secretKey("your-secret-key") + .projectName("my-project") + .batchSize(512) + .exportIntervalMs(5000) + .build() + ); + + FITracer tracer = TraceAI.getTracer(); + ``` + + | Builder method | Default | Description | + |----------------|---------|-------------| + | `baseUrl(String)` | `FI_BASE_URL` env var | Backend endpoint | + | `apiKey(String)` | `FI_API_KEY` env var | API authentication | + | `secretKey(String)` | `FI_SECRET_KEY` env var | Secondary authentication | + | `projectName(String)` | `FI_PROJECT_NAME` env var | Project identifier | + | `serviceName(String)` | project name | OpenTelemetry service name | + | `hideInputs(boolean)` | false | Suppress input values | + | `hideOutputs(boolean)` | false | Suppress output values | + | `hideInputMessages(boolean)` | false | Suppress input messages | + | `hideOutputMessages(boolean)` | false | Suppress output messages | + | `enableConsoleExporter(boolean)` | false | Log spans to console | + | `batchSize(int)` | 512 | Span batch size | + | `exportIntervalMs(long)` | 5000 | Export interval in ms | + + For **Spring Boot**, add the starter dependency and configure via `application.yml`: + + ```yaml + traceai: + enabled: true + base-url: https://api.futureagi.com + api-key: ${FI_API_KEY} + secret-key: ${FI_SECRET_KEY} + project-name: my-app + batch-size: 512 + export-interval-ms: 5000 + ``` + + The `FITracer` bean is auto-created and available for injection. + + + ```csharp + using FIInstrumentation; + using FIInstrumentation.Types; + + var tracer = TraceAI.Register(opts => + { + opts.ProjectName = "my-project"; + opts.ProjectType = ProjectType.Observe; + opts.Transport = Transport.Http; + opts.Batch = true; + opts.Verbose = true; + opts.TraceConfig = TraceConfig.Builder() + .HideInputs(false) + .HideOutputs(false) + .Build(); + }); + ``` + + | Property | Type | Default | Description | + |----------|------|---------|-------------| + | `ProjectName` | string | `FI_PROJECT_NAME` env var | Project identifier | + | `ProjectType` | ProjectType | Experiment | `Experiment` or `Observe` | + | `ProjectVersionName` | string | null | Version label (Experiment only) | + | `EvalTags` | List<EvalTag> | null | Evaluation configs (Experiment only) | + | `Metadata` | Dictionary | null | Custom metadata | + | `Batch` | bool | true | Use batch span processor | + | `SetGlobalTracerProvider` | bool | true | Register as global provider | + | `Transport` | Transport | Http | `Http` or `Grpc` | + | `ApiKey` | string | `FI_API_KEY` env var | API key | + | `SecretKey` | string | `FI_SECRET_KEY` env var | Secret key | + | `TraceConfig` | TraceConfig | null | Privacy/masking configuration | + | `EnableConsoleExporter` | bool | false | Log spans to console | + | `Verbose` | bool | true | Print config on startup | + + **Returns:** `FITracer` - use for creating custom spans. + + + +## ProjectType + +| Value | Use for | +|-------|---------| +| `EXPERIMENT` | Development and testing. Supports eval tags and version names. | +| `OBSERVE` | Production monitoring. No eval tags, no version names. | + +## SemanticConvention (Python/TypeScript) + +Controls how span attributes are named. We recommend `OTEL_GENAI` for standard OpenTelemetry GenAI conventions. + +| Value | Attribute prefix | Use for | +|-------|-----------------|---------| +| `OTEL_GENAI` | `gen_ai.*` | Recommended - OpenTelemetry GenAI standard | +| `FI` | `fi.*` | Legacy Future AGI format (default) | +| `OPENINFERENCE` | `openinference.*` | Arize Phoenix compatibility | +| `OPENLLMETRY` | `traceloop.*` | Traceloop / OpenLLMetry compatibility | + + + Pass `semantic_convention=SemanticConvention.OTEL_GENAI` for the best interoperability with other OpenTelemetry tools. + + diff --git a/src/pages/docs/observe/features/manual-tracing/semantic-conventions.mdx b/src/pages/docs/sdk/tracing/semantic-conventions.mdx similarity index 99% rename from src/pages/docs/observe/features/manual-tracing/semantic-conventions.mdx rename to src/pages/docs/sdk/tracing/semantic-conventions.mdx index ebbcec89..e91d5891 100644 --- a/src/pages/docs/observe/features/manual-tracing/semantic-conventions.mdx +++ b/src/pages/docs/sdk/tracing/semantic-conventions.mdx @@ -868,13 +868,13 @@ Every LLM provider returns data in a different format. Without a standard set of ## Next Steps - + Attach custom data, tags, session IDs, and prompt templates to spans. - + Use FITracer decorators and context managers for typed spans. - + Register a tracer provider and add instrumentation. diff --git a/src/pages/docs/observe/features/manual-tracing/set-session-user-id.mdx b/src/pages/docs/sdk/tracing/set-session-user-id.mdx similarity index 97% rename from src/pages/docs/observe/features/manual-tracing/set-session-user-id.mdx rename to src/pages/docs/sdk/tracing/set-session-user-id.mdx index 91f4a21f..72735aa0 100644 --- a/src/pages/docs/observe/features/manual-tracing/set-session-user-id.mdx +++ b/src/pages/docs/sdk/tracing/set-session-user-id.mdx @@ -309,16 +309,16 @@ Traces are isolated by default. Without a session or user identifier, there is n ## Next Steps - + Register a tracer provider and add instrumentation. - + Attach custom data to spans for filtering and evals. - + Use FITracer decorators and context managers for typed spans. - + Redact sensitive data with TraceConfig before export. diff --git a/src/pages/docs/observe/features/manual-tracing/set-up-tracing.mdx b/src/pages/docs/sdk/tracing/set-up-tracing.mdx similarity index 97% rename from src/pages/docs/observe/features/manual-tracing/set-up-tracing.mdx rename to src/pages/docs/sdk/tracing/set-up-tracing.mdx index 20f8fb08..df74003b 100644 --- a/src/pages/docs/observe/features/manual-tracing/set-up-tracing.mdx +++ b/src/pages/docs/sdk/tracing/set-up-tracing.mdx @@ -308,19 +308,19 @@ Tracing captures every LLM call, tool invocation, or custom operation in your ap Browse all supported framework instrumentors. - + Use TraceAI helpers for sessions, users, and context. - + Attach custom data to spans for filtering and evals. - + Group traces into sessions and link them to end users. - + Redact sensitive data with TraceConfig before export. - + Register an Observe project and start capturing traces. diff --git a/src/pages/docs/sdk/tracing/trace-config.mdx b/src/pages/docs/sdk/tracing/trace-config.mdx new file mode 100644 index 00000000..eeecdb28 --- /dev/null +++ b/src/pages/docs/sdk/tracing/trace-config.mdx @@ -0,0 +1,124 @@ +--- +title: "TraceConfig" +description: "Configure masking and PII redaction with TraceConfig." +--- + + +Control what data gets captured. Useful for privacy compliance, reducing payload size, or masking sensitive data. + + + + ```python + from fi_instrumentation import TraceConfig + + config = TraceConfig( + hide_inputs=True, + hide_outputs=True, + pii_redaction=True, + ) + + # Pass to instrumentors + OpenAIInstrumentor().instrument( + tracer_provider=trace_provider, + config=config, + ) + ``` + + + ```java + TraceAI.init(TraceConfig.builder() + .baseUrl("https://api.futureagi.com") + .apiKey("your-key") + .projectName("my-project") + .hideInputs(true) + .hideOutputs(true) + .hideInputMessages(true) + .hideOutputMessages(true) + .build() + ); + ``` + + + In TypeScript, `TraceConfig` is passed per-instrumentor, not to `register()`: + + ```typescript + import { OpenAIInstrumentation } from "@traceai/openai"; + import { registerInstrumentations } from "@opentelemetry/instrumentation"; + + registerInstrumentations({ + tracerProvider, + instrumentations: [ + new OpenAIInstrumentation({ + traceConfig: { + hideInputs: true, + hideOutputs: true, + hideInputImages: true, + hideEmbeddingVectors: true, + base64ImageMaxLength: 16000, + piiRedaction: true, + }, + }), + ], + }); + ``` + + + ```csharp + var tracer = TraceAI.Register(opts => + { + opts.ProjectName = "my-project"; + opts.TraceConfig = TraceConfig.Builder() + .HideInputs(true) + .HideOutputs(true) + .HideInputImages(true) + .HideEmbeddingVectors(true) + .Base64ImageMaxLength(16000) + .Build(); + }); + ``` + + + +| Field | Type | Default | What it hides | +|-------|------|---------|--------------| +| `hide_inputs` | bool | False | All input values and messages | +| `hide_outputs` | bool | False | All output values and messages | +| `hide_input_messages` | bool | False | Input messages only | +| `hide_output_messages` | bool | False | Output messages only | +| `hide_input_images` | bool | False | Images in inputs | +| `hide_input_text` | bool | False | Text in input messages | +| `hide_output_text` | bool | False | Text in output messages | +| `hide_embedding_vectors` | bool | False | Embedding vectors | +| `hide_llm_invocation_parameters` | bool | False | Model parameters (temperature, etc.) | +| `base64_image_max_length` | int | 32000 | Truncate base64 images beyond this length | +| `pii_redaction` | bool | False | Automatically mask PII (Python only) | + +Each field maps to an environment variable with the `FI_` prefix (e.g. `hide_inputs` -> `FI_HIDE_INPUTS`). + +## PII Redaction (Python) + +When `pii_redaction=True`, the SDK automatically detects and masks 6 types of personally identifiable information: + +| PII Type | Pattern | Replaced with | +|----------|---------|--------------| +| Email addresses | `user@example.com` | `` | +| Social Security Numbers | `123-45-6789` | `` | +| Credit card numbers | `4111-1111-1111-1111` | `` | +| API keys | `sk_live_...`, `pk_test_...` | `` | +| IP addresses (IPv4) | `192.168.1.1` | `` | +| Phone numbers | `+1-555-123-4567` | `` | + +```python +# Enable via code +config = TraceConfig(pii_redaction=True) + +# Or via environment variable +# export FI_PII_REDACTION=true + +# Direct usage +from fi_instrumentation.instrumentation.pii_redaction import redact_pii_in_string + +redacted = redact_pii_in_string("Email me at test@example.com") +# "Email me at " +``` + diff --git a/src/pages/docs/self-hosting.mdx b/src/pages/docs/self-hosting.mdx index dd3c9fbf..28ddd24f 100644 --- a/src/pages/docs/self-hosting.mdx +++ b/src/pages/docs/self-hosting.mdx @@ -1,49 +1,28 @@ --- -title: "Self-Hosting Future AGI: Deploy on Your Own Infrastructure" -description: "Deploy the full Future AGI platform on your own infrastructure using Docker Compose. Follow the step-by-step guide to get all services running locally." +title: "Self-hosting Future AGI" +description: "Run the entire Future AGI platform on your own infrastructure with Docker Compose. Your traces, datasets, evaluations, and model calls stay inside your network" --- -## About +Future AGI is fully open-source. Self-hosting runs the **entire stack on your own machines**, so all traces, datasets, evaluations, and model calls stay within your network. The backend is Django, the frontend is React + Vite, and the LLM gateway is Go, all deployed together with Docker Compose -Future AGI is fully open-source. Self-hosting runs the entire stack on your machines — all traces, datasets, evaluations, and model calls stay within your network. Backend is Django, frontend is React + Vite, LLM gateway is Go. +## When to self-host -Not sure if you need this? The hosted version at [app.futureagi.com](https://app.futureagi.com) is easier to operate. Self-host when you need **data residency**, **air-gapped environments**, **cost control at scale**, or **deep customization**. +The [**cloud hosted version**](https://app.futureagi.com) is the easiest way to run Future AGI, with nothing to operate. Self-host when you need: -## Quick start +- **Data residency**: keep all data inside your own network +- **Air-gapped environments**: run with no outbound dependencies +- **Cost control at scale**: own the infrastructure +- **Deep customization**: modify the open-source stack to fit your needs -```bash -git clone https://github.com/future-agi/future-agi.git -cd future-agi -cp .env.example .env -docker pull futureagi/future-agi:v1.8.19_base -docker compose up -``` - -First boot builds from source (~10–15 min). After `Application startup complete`: - -| Service | URL | -|---|---| -| Frontend | http://localhost:3000 | -| Backend API | http://localhost:8000 | -| PeerDB UI | http://localhost:3001 — `peerdb` / `peerdb` | - -## Deployment options - -| Option | Status | -|---|---| -| Docker Compose | Available | -| Helm / Kubernetes | Coming soon | -| Air-gapped | Coming soon | +## What you deploy -## Architecture - -21 containers across four layers. +Self-hosting brings up the full platform (around **21 containers, with no external dependencies**) across four layers: ``` Browser └─ frontend (React/nginx) - └─ backend (Django) ──── gateway (Go) ──── OpenAI · Anthropic · Gemini · Bedrock - ├── postgres primary DB + WAL replication + └─ backend (Django) ──── gateway (Go) ──── OpenAI · Anthropic · Gemini · Bedrock + ├── postgres primary database ├── clickhouse analytics store ├── redis cache / pub-sub ├── minio object storage @@ -52,51 +31,31 @@ Browser postgres ──── PeerDB CDC ──── clickhouse (continuous replication) ``` -**Application** — `frontend` · `backend` · `worker` · `gateway` · `serving` · `code-executor` - -**Data** — `postgres` · `clickhouse` · `redis` · `minio` +- **Application**: `frontend`, `backend`, `worker`, `gateway`, `serving`, `code-executor` +- **Data**: `postgres`, `clickhouse`, `redis`, `minio` +- **Workflow**: `temporal` +- **CDC**: PeerDB (continuous Postgres → ClickHouse replication) -**Workflow** — `temporal` +Everything runs on your machines; nothing leaves your network. The full service-by-service breakdown lives in [Configure](/docs/self-hosting/configuration). -**CDC (PeerDB)** — `peerdb-catalog` · `peerdb-temporal` · `peerdb-minio` · `peerdb-flow-api` · `peerdb-flow-worker` · `peerdb-flow-snapshot-worker` · `peerdb-server` · `peerdb-ui` · `peerdb-temporal-init` · `peerdb-init` +## Deployment options -| Layer | Service | Purpose | -|---|---|---| -| App | `frontend` | React SPA served by nginx | -| App | `backend` | Django REST + gRPC + WebSocket API | -| App | `worker` | Temporal worker — evals, agent loops, data jobs | -| App | `gateway` | Go LLM proxy — routing, retries, rate limits, logging | -| App | `serving` | Embeddings and small model inference | -| App | `code-executor` | nsjail-sandboxed eval code runner (`privileged: true` required) | -| Data | `postgres` | Primary DB — users, traces, datasets, evals, prompts | -| Data | `clickhouse` | Analytics DB — replicated from Postgres via PeerDB | -| Data | `redis` | Cache, rate limits, WebSocket pub/sub | -| Data | `minio` | S3-compatible object storage (swap for S3 in prod) | -| Workflow | `temporal` | Durable workflow engine — shares main Postgres | -| CDC | PeerDB stack | Continuous Postgres → ClickHouse replication (10 services) | +| Option | Status | +|---|---| +| Docker Compose | Available | +| Helm / Kubernetes | Coming soon | +| Air-gapped | Coming soon | -## Next Steps +## Where to go next - - - Hardware tiers, platform compatibility, ports reference. - - - Setup, deployment modes, day-to-day operations. - - - Full `.env` reference — secrets, ports, flags, keys. - - - LLM gateway providers, PeerDB mirrors, Temporal workers. - - - Create accounts via email or Django shell. + + + System requirements, prerequisites, environment variables, and setup - - Hardening, backups, monitoring, upgrades. + + Fixes for common errors and answers to frequent questions - - Solutions for every known error. + + Get help from the Future AGI team and community diff --git a/src/pages/docs/self-hosting/configuration/environment.mdx b/src/pages/docs/self-hosting/configuration/environment.mdx new file mode 100644 index 00000000..e978151e --- /dev/null +++ b/src/pages/docs/self-hosting/configuration/environment.mdx @@ -0,0 +1,130 @@ +--- +title: "Environment variables" +description: "Complete .env reference for a self-hosted Future AGI instance" +--- + +## In this page + +Every setting the stack reads at boot comes from a single `.env` file in the repo root. This page is the complete reference, grouped by what each variable does. The stack boots fine with the shipped defaults. The only thing you *must* change before sharing the instance is the `CHANGEME` secrets. + +```bash +cp .env.example .env +``` + + +Doing a local trial? Skip to [Installation](/docs/self-hosting/installation). The defaults work as-is. Come back here when you're ready to set secrets, add LLM provider keys, or turn on email. + + +## Required Secrets + +Replace every `CHANGEME` in this group before anyone else can reach the instance. Generate each value with the command shown. + +| Variable | Generate with | Used by | +|---|---|---| +| `SECRET_KEY` | `openssl rand -hex 32` | Django sessions, CSRF, password reset | +| `PG_PASSWORD` | `openssl rand -base64 24` | PostgreSQL auth | +| `MINIO_ROOT_PASSWORD` | `openssl rand -base64 24` | MinIO object storage auth | +| `AGENTCC_INTERNAL_API_KEY` | `openssl rand -hex 32` | Backend and gateway shared secret | + + +`PG_PASSWORD` is written to the Postgres volume on **first boot only**. If you change it after the volume exists, authentication fails. See the fix in [Troubleshooting](/docs/self-hosting/troubleshooting). Set it before your first `docker compose up`. + + +## Database Credentials + +| Variable | Default | Notes | +|---|---|---| +| `PG_USER` | `futureagi` | PostgreSQL username | +| `PG_PASSWORD` | `CHANGEME` | **Must change** | +| `PG_DB` | `futureagi` | PostgreSQL database name | +| `MINIO_ROOT_USER` | `futureagi` | MinIO username | +| `MINIO_ROOT_PASSWORD` | `CHANGEME` | **Must change** | +| `CH_USE_REPLICATED_ENGINES` | `false` | `true` only for multi-node ClickHouse | + +## Ports + +Every service port is configurable. The full table (defaults, what each binds to, and exposure scope) lives in [Requirements](/docs/self-hosting/requirements#network-ports), so you can plan firewall rules in one place. + +## Backend Runtime + +| Variable | Default | Description | +|---|---|---| +| `ENV_TYPE` | `development` | One of `development`, `staging`, or `prod`. Prod mode disables debug output and enables `check --deploy` | +| `FAST_STARTUP` | `false` | Skip migrations on restart (dev only). Always `false` in production | +| `GRANIAN_WORKERS` | `1` | ASGI worker processes. Set to your CPU count in production | +| `GRANIAN_THREADS` | `2` | Threads per worker | +| `ENABLE_GRPC` | `true` | Enable the gRPC endpoint | +| `ENABLE_HTTP` | `true` | Enable the HTTP/REST endpoint | + +## Temporal Worker + +| Variable | Default | Description | +|---|---|---| +| `TEMPORAL_NAMESPACE` | `default` | Temporal namespace | +| `TEMPORAL_ALL_QUEUES` | `true` | Single worker polls all queues. Set `false` and use the dev overlay for per-queue workers | +| `TEMPORAL_MAX_CONCURRENT_ACTIVITIES` | `50` | Max concurrent activity tasks | +| `TEMPORAL_MAX_CONCURRENT_WORKFLOW_TASKS` | `50` | Max concurrent workflow tasks | + +Tuning guidance lives in [System configuration](/docs/self-hosting/configuration/system#temporal-workers). + +## LLM Gateway + +| Variable | Default | Description | +|---|---|---| +| `AGENTCC_INTERNAL_API_KEY` | `CHANGEME` | **Must change.** The backend authenticates gateway calls with this shared secret | + +Setting a key here is only half the job. The gateway also needs a `config.yaml` listing the providers it may route to. See [System configuration](/docs/self-hosting/configuration/system#llm-gateway). + +## LLM Provider Keys + +Set a key for each provider you'll use and leave the rest blank. These are read by the gateway via `${VAR}` interpolation in `config.yaml`. + +| Variable | Provider | +|---|---| +| `OPENAI_API_KEY` | OpenAI | +| `ANTHROPIC_API_KEY` | Anthropic | +| `GOOGLE_API_KEY` | Google Gemini | +| `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / `AWS_REGION` | AWS Bedrock + S3 | + +## Email (Mailgun) + +Email delivery powers self-service sign-up and password reset. Without it, you create users from the Django shell during [Installation](/docs/self-hosting/installation). Set these to turn on the email flow: + +| Variable | Description | +|---|---| +| `MAILGUN_API_KEY` | Mailgun private API key | +| `MAILGUN_SENDER_DOMAIN` | Verified Mailgun sending domain | +| `DEFAULT_FROM_EMAIL` | `From:` address for outbound email | +| `SERVER_EMAIL` | `From:` address for Django admin error email | + +## Frontend Build-Time + + +These are baked into the JavaScript bundle at Vite build time. Changing them requires a rebuild: `docker compose build frontend`. + + +| Variable | Default | Description | +|---|---|---| +| `VITE_HOST_API` | `http://localhost:8000` | Backend URL as seen by the browser. In production, use your public backend URL | +| `VITE_ENVIRONMENT` | `development` | Frontend analytics and feature flags | + +## Optional + +| Variable | Default | Description | +|---|---|---| +| `RECAPTCHA_ENABLED` | `false` | Enable reCAPTCHA on registration | +| `RECAPTCHA_SECRET_KEY` | `(none)` | reCAPTCHA v2/v3 server-side key | +| `VITE_GOOGLE_SITE_KEY` | `(none)` | reCAPTCHA client-side key (requires a frontend rebuild) | +| `FUTURE_AGI_CLOUD_API_KEY` | `(none)` | Enterprise-tier Cloud features only. Leave blank for the open-source build | +| `FUTURE_AGI_CLOUD_API_URL` | `https://api.futureagi.com` | Do not change | + +## Dive deeper + + + + Point the LLM gateway at your providers and set up PeerDB mirrors + + + Harden the instance before exposing it to users + + diff --git a/src/pages/docs/self-hosting/configuration/system.mdx b/src/pages/docs/self-hosting/configuration/system.mdx new file mode 100644 index 00000000..b28489dc --- /dev/null +++ b/src/pages/docs/self-hosting/configuration/system.mdx @@ -0,0 +1,143 @@ +--- +title: "System configuration" +description: "Configure the LLM gateway, PeerDB replication, and Temporal workers for your self-hosted instance." +--- + +## In this page + +A few parts of the stack are configured outside `.env`: the LLM gateway needs a `config.yaml` listing its providers, PeerDB needs its replication mirrors running, and Temporal workers can be tuned for throughput. This page covers all three. Set your secrets and provider keys in [Environment Variables](/docs/self-hosting/configuration/environment) first, since the config here references them. + +## LLM Gateway + +The gateway is a Go proxy that routes every model call the platform makes. It reads a `config.yaml` that lists which providers it may use and which models each exposes. + + +Model calls fail until this file exists. The gateway ships with `config.example.yaml` (OpenAI enabled) but **not** a live `config.yaml`. You create one in the steps below. + + + + +```bash +cp agentcc-gateway/config.example.yaml \ + agentcc-gateway/config.yaml +``` + + + +Edit `config.yaml`: uncomment the providers you want and reference their keys with `${VAR}` interpolation. Set the matching keys (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, …) in `.env`. See the provider examples below. + + + +Point the gateway volume at your `config.yaml` in `docker-compose.yml`: + +```yaml +volumes: + - ./agentcc-gateway/config.yaml:/app/config.yaml:ro +``` + +```bash +docker compose up -d --force-recreate gateway +``` + + + + +`config.yaml` is gitignored and holds live API keys. Treat it as a secret. Never commit it. + + +### Provider Examples + + + +```yaml +providers: + openai: + api_key: "${OPENAI_API_KEY}" + api_format: "openai" + models: [gpt-4o, gpt-4o-mini] + + anthropic: + api_key: "${ANTHROPIC_API_KEY}" + api_format: "anthropic" + models: [claude-opus-4-5, claude-sonnet-4-5] + + gemini: + api_key: "${GOOGLE_API_KEY}" + api_format: "gemini" + models: [gemini-2.0-flash, gemini-1.5-pro] +``` + + +```yaml +providers: + bedrock: + api_key: "${AWS_SECRET_ACCESS_KEY}" + api_format: "bedrock" + region: "${AWS_REGION}" + access_key: "${AWS_ACCESS_KEY_ID}" + models: [anthropic.claude-3-5-sonnet-20241022-v2:0] +``` + + +```yaml +providers: + vertex: + base_url: "https://us-central1-aiplatform.googleapis.com" + api_key: "${GOOGLE_ACCESS_TOKEN}" + api_format: "gemini" + headers: + x-gcp-project: "${GCP_PROJECT_ID}" + x-gcp-location: "us-central1" + models: [gemini-2.0-flash-001] +``` + +Vertex uses a Bearer token, not a static API key. Rotate `GOOGLE_ACCESS_TOKEN` with a sidecar that calls `gcloud auth print-access-token`. + + + +For routing rules, rate limits, caching, and the full config reference, see [Agent Command Center → Self-Hosted](/docs/command-center/deployment/self-hosted). + +## PeerDB Replication + +PeerDB continuously replicates Postgres tables into ClickHouse (change-data-capture) so trace and eval analytics stay fast. It runs on its own. The only thing you typically touch is a first-boot timing fix. + + +**First-boot timing.** `peerdb-init` runs the moment the stack starts, sometimes before Django has finished its migrations. If mirrors show "not started" in the PeerDB UI, re-run init once the backend is up: + +```bash +docker compose logs -f backend # wait for "Application startup complete" +docker compose run --rm peerdb-init bash /setup.sh # re-run init +``` + + +Verify at [http://localhost:3001](http://localhost:3001). Mirrors should move to `running` within seconds. Re-run the same init command after any upgrade that changes replicated tables. + +## Temporal Workers + +Temporal runs the platform's background jobs and evaluation pipelines. How those jobs are distributed across workers depends on one flag. + +**All-queue (default).** One worker polls every task queue. Controlled by `TEMPORAL_ALL_QUEUES=true` in `.env`. This is the right setup for most self-hosted deployments. + +**Per-queue (dev overlay).** Six dedicated workers, one per queue, brought up by the [dev overlay](/docs/self-hosting/installation#other-ways-to-run-it): + +| Service | Queue | Typical concurrency | +|---|---|---| +| `worker-default` | `default` | 100 | +| `worker-tasks-s` | `tasks_s` | 200 | +| `worker-tasks-l` | `tasks_l` | 50 | +| `worker-tasks-xl` | `tasks_xl` | 10 | +| `worker-trace-ingestion` | `trace_ingestion` | 100 | +| `worker-agent-compass` | `agent_compass` | 50 | + +Tune throughput with `TEMPORAL_MAX_CONCURRENT_ACTIVITIES` and `TEMPORAL_MAX_CONCURRENT_WORKFLOW_TASKS` in `.env`. The Temporal UI is available in dev mode at [http://localhost:8085](http://localhost:8085). + +## Dive Deeper + + + + Hardening, backups, and monitoring before going live. + + + Fixes for gateway, PeerDB, and Temporal errors. + + diff --git a/src/pages/docs/self-hosting/docker-compose.mdx b/src/pages/docs/self-hosting/docker-compose.mdx index 1ce057d7..c039421e 100644 --- a/src/pages/docs/self-hosting/docker-compose.mdx +++ b/src/pages/docs/self-hosting/docker-compose.mdx @@ -23,7 +23,7 @@ First boot builds from source (~10–15 min). When the backend logs `Application - **Backend API** — [http://localhost:8000](http://localhost:8000) - **PeerDB UI** — [http://localhost:3001](http://localhost:3001) · `peerdb` / `peerdb` -Replace `CHANGEME` secrets in `.env` before sharing the instance with others. See [Environment Variables](/docs/self-hosting/environment). +Replace `CHANGEME` secrets in `.env` before sharing the instance with others. See [Environment Variables](/docs/self-hosting/configuration/environment). --- @@ -112,7 +112,7 @@ docker compose run --rm peerdb-init bash /setup.sh ## Next Steps - + Configure secrets, ports, and runtime flags in `.env`. diff --git a/src/pages/docs/self-hosting/environment.mdx b/src/pages/docs/self-hosting/environment.mdx deleted file mode 100644 index 0967ef80..00000000 --- a/src/pages/docs/self-hosting/environment.mdx +++ /dev/null @@ -1,118 +0,0 @@ ---- -title: "Self-Hosting Environment Variables" -description: "Full .env reference for self-hosted Future AGI — secrets, database credentials, runtime flags, LLM provider keys, email, and frontend build-time configuration." ---- - -## About - -Reference for every environment variable the stack reads from `.env`. Grouped by purpose: secrets, database credentials, runtime flags, LLM provider keys, email, and frontend build-time config. - -```bash -cp .env.example .env -``` - -The stack boots fine with defaults. Replace `CHANGEME` secrets before sharing with others. - -## Required secrets - -| Variable | Generate with | Used by | -|---|---|---| -| `SECRET_KEY` | `openssl rand -hex 32` | Django sessions, CSRF, password reset | -| `PG_PASSWORD` | `openssl rand -base64 24` | PostgreSQL auth | -| `MINIO_ROOT_PASSWORD` | `openssl rand -base64 24` | MinIO object storage auth | -| `AGENTCC_INTERNAL_API_KEY` | `openssl rand -hex 32` | Backend ↔ gateway shared secret | - -## Database credentials - -| Variable | Default | Notes | -|---|---|---| -| `PG_USER` | `futureagi` | PostgreSQL username | -| `PG_PASSWORD` | `CHANGEME` | **Must change** | -| `PG_DB` | `futureagi` | PostgreSQL database name | -| `MINIO_ROOT_USER` | `futureagi` | MinIO username | -| `MINIO_ROOT_PASSWORD` | `CHANGEME` | **Must change** | -| `CH_USE_REPLICATED_ENGINES` | `false` | `true` only for multi-node ClickHouse | - -## Ports - -All configurable. See [Requirements → Ports reference](/docs/self-hosting/requirements#ports-reference) for the full table with defaults and exposure scope. - -## Backend runtime - -| Variable | Default | Description | -|---|---|---| -| `ENV_TYPE` | `development` | `development` · `staging` · `prod` — prod mode disables debug output, enables `check --deploy` | -| `FAST_STARTUP` | `false` | Skip migrations on restart (dev only). Always `false` in production. | -| `GRANIAN_WORKERS` | `1` | ASGI worker processes. Set to CPU count in production. | -| `GRANIAN_THREADS` | `2` | Threads per worker. | -| `ENABLE_GRPC` | `true` | Enable gRPC endpoint. | -| `ENABLE_HTTP` | `true` | Enable HTTP/REST endpoint. | - -## Temporal worker - -| Variable | Default | Description | -|---|---|---| -| `TEMPORAL_NAMESPACE` | `default` | Temporal namespace. | -| `TEMPORAL_ALL_QUEUES` | `true` | Single worker polls all queues. Set `false` + use dev overlay for per-queue workers. | -| `TEMPORAL_MAX_CONCURRENT_ACTIVITIES` | `50` | Max concurrent activity tasks. | -| `TEMPORAL_MAX_CONCURRENT_WORKFLOW_TASKS` | `50` | Max concurrent workflow tasks. | - -## LLM gateway - -| Variable | Default | Description | -|---|---|---| -| `AGENTCC_INTERNAL_API_KEY` | `CHANGEME` | **Must change.** Backend authenticates gateway calls with this. | - -## LLM provider keys - -Leave blank for providers you're not using. - -| Variable | Provider | -|---|---| -| `OPENAI_API_KEY` | OpenAI | -| `ANTHROPIC_API_KEY` | Anthropic | -| `GOOGLE_API_KEY` | Google Gemini | -| `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / `AWS_REGION` | AWS Bedrock + S3 | - -## Email (Mailgun) - -Required for email-based sign-up and password reset. Without these, create users via the Django shell — see [User Management](/docs/self-hosting/user-management). - -| Variable | Description | -|---|---| -| `MAILGUN_API_KEY` | Mailgun private API key | -| `MAILGUN_SENDER_DOMAIN` | Verified Mailgun sending domain | -| `DEFAULT_FROM_EMAIL` | `From:` address for outbound emails | -| `SERVER_EMAIL` | Django admin error emails | - -## Frontend build-time - - -These are baked into the JS bundle at Vite build time. Changing them requires rebuilding: `docker compose build frontend` - - -| Variable | Default | Description | -|---|---|---| -| `VITE_HOST_API` | `http://localhost:8000` | Backend URL as seen by the browser. In production: your public backend URL. | -| `VITE_ENVIRONMENT` | `development` | Frontend analytics and feature flags. | - -## Optional - -| Variable | Default | Description | -|---|---|---| -| `RECAPTCHA_ENABLED` | `false` | Enable reCAPTCHA on registration. | -| `RECAPTCHA_SECRET_KEY` | — | reCAPTCHA v2/v3 server-side key. | -| `VITE_GOOGLE_SITE_KEY` | — | reCAPTCHA client-side key (requires frontend rebuild). | -| `FUTURE_AGI_CLOUD_API_KEY` | — | EE-tier Cloud features only. Leave blank for OSS. | -| `FUTURE_AGI_CLOUD_API_URL` | `https://api.futureagi.com` | Do not change. | - -## Next Steps - - - - Set up LLM gateway providers and PeerDB mirrors. - - - Hardening checklist for exposing the stack to users. - - diff --git a/src/pages/docs/self-hosting/installation.mdx b/src/pages/docs/self-hosting/installation.mdx new file mode 100644 index 00000000..9bc66c6d --- /dev/null +++ b/src/pages/docs/self-hosting/installation.mdx @@ -0,0 +1,140 @@ +--- +title: "Installation" +description: "Install a self-hosted Future AGI instance with Docker Compose." +--- + +Docker Compose is the supported way to run a self-hosted Future AGI instance. + +## In this page + +Confirm your host meets the [requirements](/docs/self-hosting/requirements) first, then `./bin/install` does the rest: + +- Bootstraps your `.env` +- Brings up the stack +- Waits for the backend health check +- Prompts you to create the first user + +First boot pulls the app images from Docker Hub and builds the small fi-collector image from source, so give it a few minutes the first time. + + +Run `git clone https://github.com/future-agi/future-agi.git && cd future-agi && ./bin/install`, then open [http://localhost:3000](http://localhost:3000). + + +## Install + + + +```bash +git clone https://github.com/future-agi/future-agi.git +cd future-agi +./bin/install # Windows: bin\install.ps1 +``` + +The stack boots fine against an empty `.env`, so you can take the defaults for a local trial. + +By default the installer brings up the standard stack (around 12 containers). Add `--full` to include the PeerDB CDC stack (around 22 containers) that populates the analytics views. + + + +The installer prompts you at the end. If you passed `--skip-user-creation`, create the account from the CLI instead: + +```bash +docker compose exec backend python manage.py create_user +``` + +You will be asked for an email, full name, and password. To script it, pass them inline: + +```bash +docker compose exec backend python manage.py create_user \ + --email you@example.com \ + --name "Your Name" \ + --password yourpassword +``` + + + +Log in at [http://localhost:3000](http://localhost:3000) with the user you just created. The backend API is at [http://localhost:8000](http://localhost:8000). + + + +### Installer flags + +| Flag | What it does | +|---|---| +| `--full` | Add the PeerDB CDC stack (around 22 containers) so the analytics views populate | +| `--skip-user-creation` | Skip the first-user prompt; create the account later with `create_user` | +| `--no-up` | Bootstrap `.env` only, without starting the stack | +| `--wipe-volumes` | Remove stale project volumes before starting (destroys existing data) | +| `--new-instance` | Start a fresh instance when existing volumes are detected | + + +**Apple Silicon and arm64 hosts.** Prebuilt images are `linux/amd64`. On M-series Macs they run under Rosetta 2 (auto-enabled on Docker Desktop 4.16+), which is fine for evaluation with a 20 to 50 percent performance cost. For native arm64, build locally with `docker compose build` instead of pulling. On Linux arm64 such as Graviton, install `qemu-user-static`. + + +## Install without the script + +The installer is a convenience wrapper, not a requirement. To run the same steps by hand: + +```bash +cp .env.example .env # optional; an empty .env works for local +docker compose up -d +``` + +Then create the first user with the same `create_user` command shown above. + +## Verify the stack + +Check that every service is healthy before you log in. Under-provisioned RAM is the most common reason the backend never finishes booting, so confirm the [requirements](/docs/self-hosting/requirements) if it stalls. + +```bash +docker compose ps # every service should read "running" or "healthy" +docker compose logs -f backend # watch for errors while it boots +curl http://localhost:8000/health/ +``` + +The instance is ready when `/health/` returns OK. That's the same check `./bin/install` polls while it waits for the backend. + +## Everyday operations + +A short reference for the commands you will use most: + +```bash +# Tail logs +docker compose logs -f backend worker + +# Shell into a container +docker compose exec backend bash +docker compose exec postgres psql -U futureagi -d futureagi + +# Stop the stack (data persists in named volumes) +./bin/uninstall # or: docker compose down + +# Wipe all data and start clean +./bin/uninstall --wipe-data # or: docker compose down -v + +# Remove everything: containers, volumes, .env, and built images +./bin/uninstall --purge +``` + +## Other ways to run it + +| Mode | Command | Use it for | +|---|---|---| +| Standard (default) | `docker compose up -d` | Local evaluation, team installs, and VM self-hosting | +| Development | `docker compose -f docker-compose.yml -f docker-compose.dev.yml up` | Contributing to Future AGI: hot reload, per-queue workers, host-accessible database ports, and the Temporal UI | +| Frontend only | `docker compose -f docker-compose.frontend.yml up -d` | Pointing a local UI at a backend that runs elsewhere | + + +For a frontend-only deploy, set `VITE_HOST_API` to the backend URL the browser can reach. It is applied when the container starts, so changing it needs only a restart of the frontend container, not a rebuild. + + +## Dive deeper + + + + Set provider keys, secrets, and runtime flags in `.env` + + + Tune the gateway, PeerDB, and Temporal workers + + diff --git a/src/pages/docs/self-hosting/production.mdx b/src/pages/docs/self-hosting/production.mdx index c5fbb682..fa37559a 100644 --- a/src/pages/docs/self-hosting/production.mdx +++ b/src/pages/docs/self-hosting/production.mdx @@ -1,137 +1,35 @@ --- -title: "Production Hardening & Operations" -description: "Production readiness checklist — replace secrets, configure TLS, set up managed data stores, run Postgres/ClickHouse/MinIO backups, and follow the upgrade runbook." +title: "Production" +description: "What to harden before a self-hosted Future AGI instance goes live" --- -## About +Everything past a local trial happens here. The default Docker Compose stack boots with placeholder secrets, no TLS, and compose-managed data stores. That's fine on a laptop. Before real traffic reaches the instance, work through the flow below in order, then keep each page as a runbook. -Run through this before exposing the stack to real users. Covers secrets, TLS, swapping in managed data stores, backup commands for Postgres/ClickHouse/MinIO, Prometheus monitoring, and the upgrade and rollback runbook. +## In this page -## Hardening checklist +Production readiness for a self-hosted instance breaks into five steps. Do them in order the first time. -**Secrets** — replace all `CHANGEME` values before going live: +**Before you go live** -```bash -openssl rand -hex 32 # SECRET_KEY, AGENTCC_INTERNAL_API_KEY -openssl rand -base64 24 # PG_PASSWORD, MINIO_ROOT_PASSWORD -``` - -**Runtime flags** in `.env`: -- `ENV_TYPE=prod` -- `FAST_STARTUP=false` -- `GRANIAN_WORKERS=` - -**TLS** — the frontend and backend don't terminate TLS. Put Caddy, nginx, or Traefik in front: - -``` -# Caddyfile (simplest — auto-issues Let's Encrypt certs) -app.yourcompany.com { reverse_proxy localhost:3000 } -api.yourcompany.com { reverse_proxy localhost:8000 } -``` - -After setting up TLS, set `VITE_HOST_API=https://api.yourcompany.com` in `.env` and rebuild: - -```bash -docker compose build frontend && docker compose up -d frontend -``` - -**Managed data stores** — for production, replace compose-managed services: - -| Replace | With | Change | -|---|---|---| -| `postgres` | RDS / Aurora / Cloud SQL | Set `PG_*` vars to managed endpoint | -| `clickhouse` | ClickHouse Cloud | Set `CH_HOST`, `CH_PORT`, etc. | -| `redis` | ElastiCache / Upstash | Set `REDIS_URL` | -| `minio` | AWS S3 | Set `S3_ENDPOINT_URL=https://s3.amazonaws.com` + AWS creds | - - -`code-executor` requires `privileged: true`. Run on EC2 / GCE instances — not Fargate or Cloud Run. - - -**Secrets manager** — use AWS Secrets Manager, HashiCorp Vault, or GCP Secret Manager instead of a plain `.env` file. - ---- - -## Backups - -### PostgreSQL - -```bash -# Backup -docker compose exec postgres \ - pg_dump -U futureagi -d futureagi --format=custom \ - > backup-$(date +%F).dump - -# Restore -docker compose exec -T postgres \ - pg_restore -U futureagi -d futureagi --clean --if-exists \ - < backup-2026-04-22.dump -``` - -Volumes: `future-agi_postgres-data` · `future-agi_clickhouse-data` · `future-agi_redis-data` · `future-agi_minio-data` · `future-agi_peerdb-catalog-data` · `future-agi_peerdb-minio-data` - -### ClickHouse - -```sql -BACKUP TABLE default.traces TO S3('s3://your-bucket/ch-backup/', 'KEY', 'SECRET'); -``` - -ClickHouse data can also be rebuilt from scratch by re-running PeerDB init since it replicates from Postgres. - -### MinIO - -```bash -mc alias set local http://localhost:9005 futureagi -mc alias set s3 https://s3.amazonaws.com -mc mirror local/ s3/your-bucket/ -``` - ---- - -## Monitoring - -Backend exposes Prometheus metrics at `http://localhost:8000/metrics`. Add a scraper: - -```yaml -# prometheus.yml -scrape_configs: - - job_name: futureagi - static_configs: - - targets: ['localhost:8000'] - metrics_path: /metrics -``` - -Key signals: backend error rate, Temporal workflow success/failure, Postgres WAL lag (PeerDB health), ClickHouse query latency, PeerDB mirror status at [localhost:3001](http://localhost:3001). - ---- - -## Upgrades - -```bash -git pull -docker compose build -docker compose up -d -``` - -Migrations run automatically. If a migration fails: `docker compose exec backend python manage.py migrate` - -If release notes mention PeerDB changes: `docker compose run --rm peerdb-init bash /setup.sh` - -**Rollback:** - -```bash -git log --oneline -5 -git checkout -docker compose build && docker compose up -d -``` + + + The go-live pass: secrets, prod runtime flags, and managed data stores + + + Terminate TLS in front of the stack and lock down secrets + + -## Next Steps +**Operating it** - - - Symptoms, causes, and fixes for common errors. + + + Back up and restore Postgres, ClickHouse, and MinIO + + + Watch the health signals the stack actually exposes - - Tune the LLM gateway, PeerDB mirrors, and Temporal workers. + + Pull a release, run migrations, and roll back safely diff --git a/src/pages/docs/self-hosting/production/backups-restore.mdx b/src/pages/docs/self-hosting/production/backups-restore.mdx new file mode 100644 index 00000000..5b0b257f --- /dev/null +++ b/src/pages/docs/self-hosting/production/backups-restore.mdx @@ -0,0 +1,70 @@ +--- +title: "Backups & restore" +description: "Back up and restore the data stores behind a self-hosted instance" +--- + +A self-hosted instance keeps state in a few stores: Postgres for application data, ClickHouse for the observability records (spans and traces), and MinIO for object storage. Redis is a cache (sessions, locks, rate limits, pub/sub), so it rebuilds on its own and doesn't need a backup. RabbitMQ holds the task queue: losing it drops in-flight background jobs, so drain it before planned downtime rather than backing it up. + +## Postgres + +Postgres holds the application data, so back it up on a schedule. Use the custom format, and pass `-T` so `docker compose exec` doesn't allocate a TTY and mangle the binary dump: + +```bash +# Backup +docker compose exec -T postgres \ + pg_dump -U futureagi -d futureagi --format=custom \ + > backup-$(date +%F).dump + +# Restore +docker compose exec -T postgres \ + pg_restore -U futureagi -d futureagi --clean --if-exists \ + < backup-2026-04-22.dump +``` + +The named volumes that hold state. The compose project name is `futureagi`, so every volume is prefixed `futureagi_`: + +| Volume | Holds | +|---|---| +| `futureagi_postgres-data` | Postgres application data | +| `futureagi_clickhouse-data` | ClickHouse spans and traces | +| `futureagi_minio-data` | MinIO objects | +| `futureagi_rabbitmq-data` | RabbitMQ task queue | +| `futureagi_redis-data` | Redis cache (rebuildable) | +| `futureagi_peerdb-catalog-data` | PeerDB replication catalog | +| `futureagi_peerdb-minio-data` | PeerDB staging objects | +| `futureagi_fi-collector-data` | fi-collector buffer | + +## ClickHouse + +ClickHouse is not just a replica anymore. Since the CH25 cutover (`CH25_DROP_LEGACY_CDC_CHAIN` defaults to `true`), the fi-collector writes `spans` straight to ClickHouse and Django dual-writes `traces`. PeerDB only rebuilds the tables it mirrors from Postgres, so if you lose ClickHouse the observability data does **not** come back from a PeerDB re-init. Back it up on its own: + +```sql +BACKUP DATABASE default TO S3('s3://your-bucket/ch-backup/', 'KEY', 'SECRET'); +``` + + +Don't rely on PeerDB init to rebuild ClickHouse. It restores the mirrored Postgres tables, not the `spans` the collector writes directly. ClickHouse needs a real backup on its own schedule. + + +## MinIO + +Mirror the MinIO bucket to S3 with the MinIO client: + +```bash +mc alias set local http://localhost:9005 futureagi +mc alias set s3 https://s3.amazonaws.com +mc mirror local/ s3/your-bucket/ +``` + +If you've already moved to [managed data stores](/docs/self-hosting/production/checklist), your provider's own backup tooling replaces these commands. + +## Dive deeper + + + + Watch store health and replication lag + + + Roll back releases without losing data + + diff --git a/src/pages/docs/self-hosting/production/checklist.mdx b/src/pages/docs/self-hosting/production/checklist.mdx new file mode 100644 index 00000000..281fd201 --- /dev/null +++ b/src/pages/docs/self-hosting/production/checklist.mdx @@ -0,0 +1,69 @@ +--- +title: "Checklist" +description: "The go-live pass before a self-hosted instance takes real traffic" +--- + +Run through this once before the stack is reachable by anyone else. Three things separate a laptop trial from a real deployment: + +- Replace the dev-only secret defaults +- Bring the stack up with the production overlay, so it refuses to boot until those secrets are set +- Move the compose-managed data stores to managed services + +## Replace the dev-only secrets + +The stack boots with dev-only placeholders baked into `docker-compose.yml`, values like `local-dev-only-not-for-production-replace-me`, and `futureagi` for the database passwords. It runs fine with them, which is the point on a laptop and the danger in production. + +What forces real secrets is the production overlay, `deploy/docker-compose.production.yml`. It re-binds each one with `${VAR:?}`, so the stack won't start until you've set them. Bring the stack up with that overlay and set, at minimum: + +- `SECRET_KEY` +- `AGENTCC_INTERNAL_API_KEY` +- `AGENTCC_ADMIN_TOKEN` +- `PG_PASSWORD` +- `MINIO_ROOT_PASSWORD` +- `RABBITMQ_USER` and `RABBITMQ_PASSWORD` +- `FRONTEND_URL` and `VITE_HOST_API` + +```bash +openssl rand -hex 32 # SECRET_KEY, AGENTCC_INTERNAL_API_KEY, AGENTCC_ADMIN_TOKEN +openssl rand -base64 24 # PG_PASSWORD, MINIO_ROOT_PASSWORD, RABBITMQ_PASSWORD +``` + + +`PG_PASSWORD` is baked into the Postgres volume on the **first** boot, so set it before your first `docker compose up`. `MINIO_ROOT_PASSWORD` is read from the environment on every boot, so that one you can change and restart. The full field list is in [Environment variables](/docs/self-hosting/configuration/environment). + + +## Switch the backend to production mode + +Set these runtime flags before going live: + +| Variable | Go-live value | Why | +|---|---|---| +| `ENV_TYPE` | `prod` | Disables debug output and runs Django `check --deploy` | +| `FAST_STARTUP` | `false` | Always applies migrations on restart | +| `GRANIAN_WORKERS` | your CPU count | One worker per core, up from the default `1` | + +## Move to managed data stores + +Compose-managed Postgres, ClickHouse, Redis, and MinIO are fine for a trial. For production, point the stack at managed services. The catch: the backend reads these hosts from **hardcoded values in the `backend` env block of `docker-compose.yml`** (`PG_HOST: postgres`, `CH_HOST: clickhouse`, `REDIS_URL: redis://redis:6379/0`, `S3_ENDPOINT_URL: http://minio:9000`), not from `.env`. Setting them in `.env` does nothing. You edit the compose file. + +| Replace | With | Edit in `docker-compose.yml` | +|---|---|---| +| `postgres` | RDS, Aurora, or Cloud SQL | `PG_HOST` / `PG_PORT` to the managed endpoint | +| `clickhouse` | ClickHouse Cloud | `CH_HOST` / `CH_PORT` and the credentials | +| `redis` | ElastiCache or Upstash | `REDIS_URL` | +| `minio` | AWS S3 | `STORAGE_BACKEND: s3` and the S3 credentials | + + +`code-executor` runs with `privileged: true`, so it can't run on ECS Fargate or Cloud Run. Put it on an EC2 or GCE instance. The platform matrix is in [Requirements](/docs/self-hosting/requirements). + + +## Dive deeper + + + + Put TLS in front of the stack and move secrets into a manager + + + Set up backups before the instance holds real data + + diff --git a/src/pages/docs/self-hosting/production/monitoring.mdx b/src/pages/docs/self-hosting/production/monitoring.mdx new file mode 100644 index 00000000..6774fd9c --- /dev/null +++ b/src/pages/docs/self-hosting/production/monitoring.mdx @@ -0,0 +1,44 @@ +--- +title: "Monitoring" +description: "Watch a self-hosted instance with the health signals it actually exposes" +--- + +The stack has no Prometheus `/metrics` endpoint yet (the fi-collector lists a metrics exporter as a TODO), so monitoring today is built from what the containers already expose: their Docker health checks, the fi-collector's admin health endpoint, and the PeerDB console. This page covers those and the signals worth watching. + +## Container health + +The data stores (Postgres, ClickHouse, Redis, RabbitMQ, MinIO, Temporal) ship Docker health checks; the application services just show `running`. Either way, `docker compose ps` is the fastest read on what's up: + +```bash +docker compose ps # STATUS shows healthy / unhealthy per service +docker stats # live CPU and memory per container +``` + +Watch memory on `clickhouse` and the Temporal `worker` first. They're the resource drivers, and an OOM there is the most common cause of a stall. + +## fi-collector health + +The fi-collector exposes an admin endpoint on `127.0.0.1:9464` (`FI_COLLECTOR_ADMIN_PORT`), which serves a health check. Hit it to confirm the collector is up: + +```bash +curl -s http://localhost:9464/healthz +``` + +## PeerDB replication + +The Postgres-to-ClickHouse pipeline has its own console at [localhost:3001](http://localhost:3001). Mirror status there tells you whether trace analytics are keeping up with Postgres. A mirror in anything other than `running` means the dashboard is reading stale. + + +A Prometheus metrics exporter is on the fi-collector roadmap. When it lands, scrape it here. Until then, the checks above are what the stack actually exposes. + + +## Dive deeper + + + + Keep the stack current without downtime + + + Symptoms, causes, and fixes for common errors + + diff --git a/src/pages/docs/self-hosting/production/security-tls.mdx b/src/pages/docs/self-hosting/production/security-tls.mdx new file mode 100644 index 00000000..51239b2a --- /dev/null +++ b/src/pages/docs/self-hosting/production/security-tls.mdx @@ -0,0 +1,59 @@ +--- +title: "Security & TLS" +description: "Terminate TLS in front of a self-hosted instance and lock down its secrets" +--- + +Neither the frontend nor the backend terminates TLS. In production you put a reverse proxy in front of the stack to handle certificates, then point the frontend at the HTTPS endpoint. This page covers both, plus where production secrets should live. + +## Terminate TLS with a reverse proxy + +Run Caddy, nginx, or Traefik in front of the stack. Caddy is the shortest path because it issues and renews Let's Encrypt certificates on its own: + +``` +# Caddyfile +app.yourcompany.com { reverse_proxy localhost:3000 } +api.yourcompany.com { reverse_proxy localhost:8000 } +``` + + + + Point the proxy at the frontend on `localhost:3000` and the backend on `localhost:8000`. The full port list is in [Requirements](/docs/self-hosting/requirements#network-ports). + + + Set `VITE_HOST_API=https://api.yourcompany.com` in `.env`. The frontend container reads it on start and writes it into `config.js`, so no rebuild is needed. + + + ```bash + docker compose up -d frontend + ``` + + + + +`VITE_HOST_API` is applied when the frontend container starts, not at build time. If the browser still calls the old host after you change it, the container wasn't recreated: rerun `docker compose up -d frontend`. + + +## Keep secrets out of the compose file + +For anything past a single trial host, store secrets in a dedicated manager instead of a plain `.env`: + +- AWS Secrets Manager +- HashiCorp Vault +- GCP Secret Manager + +Rotate the dev-only default secrets from the [Checklist](/docs/self-hosting/production/checklist) first, then move them into the manager and inject them at deploy time. + +## Isolate the code executor + +`code-executor` runs with `privileged: true` so it can sandbox evaluation code. Keep it on a host you control, an EC2 or GCE instance, never a managed-container platform that can't grant that flag. + +## Dive deeper + + + + Protect the data behind the proxy + + + Tune the gateway, PeerDB, and Temporal workers + + diff --git a/src/pages/docs/self-hosting/production/upgrades-rollback.mdx b/src/pages/docs/self-hosting/production/upgrades-rollback.mdx new file mode 100644 index 00000000..4c2e382b --- /dev/null +++ b/src/pages/docs/self-hosting/production/upgrades-rollback.mdx @@ -0,0 +1,59 @@ +--- +title: "Upgrades & rollback" +description: "Pull a new release, run migrations, and roll back when one goes wrong" +--- + +Upgrades are a git pull and a rebuild, and migrations run automatically on boot. This page covers the routine upgrade, the two cases that need a manual step, and how to roll back. + +## Upgrade to a new release + + + + ```bash + git pull + docker compose build + docker compose up -d + ``` + + + Database migrations run automatically on backend startup. If one fails, run it by hand: + ```bash + docker compose exec backend python manage.py migrate + ``` + + + When a release changes which Postgres tables are mirrored, re-run init. The container's entrypoint is already `bash /setup.sh`, so no arguments are needed: + ```bash + docker compose run --rm peerdb-init + ``` + + + + +PeerDB init only rebuilds the tables it mirrors from Postgres. It does **not** restore the `spans` the fi-collector writes straight to ClickHouse, so it is not a recovery path for lost ClickHouse data. For that, restore from a [ClickHouse backup](/docs/self-hosting/production/backups-restore). + + +## Roll back a bad release + +Roll back to the previous commit and rebuild: + +```bash +git log --oneline -5 +git checkout +docker compose build && docker compose up -d +``` + + +Checking out older code does not undo a migration that already ran. If a release applied a migration you need to reverse, roll it back before you switch code, or restore Postgres from a backup. + + +## Dive deeper + + + + Symptoms, causes, and fixes for common errors + + + Where to get help when you're stuck + + diff --git a/src/pages/docs/self-hosting/requirements.mdx b/src/pages/docs/self-hosting/requirements.mdx index 4e0ce3be..2589a712 100644 --- a/src/pages/docs/self-hosting/requirements.mdx +++ b/src/pages/docs/self-hosting/requirements.mdx @@ -1,24 +1,36 @@ --- -title: "Self-Hosting Requirements" -description: "Hardware sizing tiers, supported platforms, OS compatibility, and network port requirements before deploying Future AGI with Docker Compose." +title: "Requirements" +description: "System requirements and support for self-hosting Future AGI." --- -## About +## In this page -Hardware tiers, supported platforms, and the network ports each service uses. Read this first to size your environment before running [Docker Compose](/docs/self-hosting/docker-compose). +Check three things before you install: + +- A host that meets the sizing for your usage +- The required software: Docker and Git +- A supported platform + +Get these right and the [Installation](/docs/self-hosting/installation) run works on the first try. + + +For a local trial: **4 CPU cores, 8 GB RAM, 20 GB disk**, Docker Engine 24+, Docker Compose v2.20+, and Git. + ## Hardware tiers +Pick the row that matches how you'll use the instance. The stack runs on the Evaluation tier, but ClickHouse and the Temporal worker are the resource drivers. Under-provisioning RAM is the most common cause of a failed first boot. + | Tier | Use case | CPU | RAM | Disk | |---|---|---|---|---| | **Evaluation** | Local trial, single user | 4 cores | 8 GB | 20 GB | -| **Team** | 1–20 users, regular eval runs | 8 cores | 16 GB | 50 GB | +| **Team** | 1-20 users, regular eval runs | 8 cores | 16 GB | 50 GB | | **Production** | 20+ users, high throughput | 16+ cores | 32+ GB | 200 GB+ SSD | -Resource drivers: ClickHouse and Temporal worker each hold ~1 GB RAM at steady state. First image build is ~6 GB disk. ClickHouse grows with trace volume; Postgres stays small. +ClickHouse and the Temporal worker each hold ~1 GB RAM at steady state. ClickHouse grows with trace volume over time; Postgres stays small. Pulling the images takes a few GB of disk on the first run. -Docker Desktop (Mac/Windows): Settings → Resources → set RAM ≥ 8 GB, disk ≥ 64 GB. The defaults (2–4 GB RAM) will OOM-kill ClickHouse or the backend. +On Docker Desktop (Mac/Windows), raise the limits in **Settings → Resources**: RAM ≥ 8 GB, disk ≥ 64 GB. The defaults (2-4 GB RAM) will OOM-kill ClickHouse or the backend before the stack finishes booting. ## Software @@ -27,70 +39,80 @@ Docker Desktop (Mac/Windows): Settings → Resources → set RAM ≥ 8 GB, disk |---|---|---| | Docker Engine | 24.0+ | `docker --version` | | Docker Compose | v2.20+ | `docker compose version` | +| Git | 2.0+ | `git --version` | +Install the tools with Homebrew, then start Colima: ```bash -brew install docker docker-compose colima +brew install docker docker-compose colima git colima start --cpu 4 --memory 8 --disk 64 ``` -Or install [Docker Desktop for Mac](https://docs.docker.com/desktop/setup/install/mac-install/) and allocate ≥ 8 GB RAM in Settings → Resources. +Install the tools with apt, then enable the Docker daemon: ```bash -sudo apt-get install -y docker.io docker-compose-v2 +sudo apt-get install -y docker.io docker-compose-v2 git sudo systemctl enable --now docker sudo usermod -aG docker $USER # log out and back in ``` -Install [Docker Desktop for Windows](https://docs.docker.com/desktop/setup/install/windows-install/) with WSL 2 backend. Allocate ≥ 8 GB RAM in Settings → Resources. +Install [Docker Desktop for Windows](https://docs.docker.com/desktop/setup/install/windows-install/) with the WSL 2 backend, then set the memory limit in WSL, not Docker's UI (the **Settings → Resources** sliders apply only to the Hyper-V backend): +```powershell +# add to %UserProfile%\.wslconfig +[wsl2] +memory=8GB +# then apply: +wsl --shutdown +``` ## Platform compatibility -The `code-executor` service requires `privileged: true`. Platforms that block it will crash the service; the rest of the stack still runs. +Future AGI runs on any host that allows **privileged containers**. The `code-executor` service needs `privileged: true` to sandbox the user code it runs for evaluations, so platforms that block privileged mode lose that one service: the rest of the stack still runs, but code-based eval features are unavailable. | Platform | Supported | Notes | |---|---|---| | Linux bare metal / EC2 / GCE / Azure VM | Yes | Full support | -| GKE / EKS with privileged enabled | Yes | Requires PodSecurityPolicy exception | +| GKE / EKS with privileged enabled | Yes | Requires a PodSecurityPolicy exception | | ECS Fargate | No | `privileged: true` not supported | | Google Cloud Run | No | Same | | Render / Railway / Fly.io | No | Managed platforms block privileged mode | -## Ports reference +Helm/Kubernetes support is on the roadmap. Docker Compose is the supported path today. + +## Network ports -All ports are configurable via `.env`. +Make sure these host ports are free before you install, or remap any that collide. Every published port reads from `.env` with a built-in default (for example `${FRONTEND_PORT:-3000}`), so you can change one without touching the Compose file. -| Service | Default | Exposed to | `.env` key | +| Service | Default | Bind | `.env` key | |---|---|---|---| | Frontend | `3000` | `0.0.0.0` | `FRONTEND_PORT` | | Backend API | `8000` | `0.0.0.0` | `BACKEND_PORT` | -| Gateway | `8090` | Internal only | `GATEWAY_PORT` | -| Model serving | `8080` | Internal only | `SERVING_PORT` | -| Code executor | `8060` | Internal only | `CODE_EXECUTOR_PORT` | -| Postgres | `5432` | `127.0.0.1` (dev: public) | `PG_PORT` | -| ClickHouse HTTP | `8123` | `127.0.0.1` (dev: public) | `CH_HTTP_PORT` | -| ClickHouse TCP | `9000` | `127.0.0.1` (dev: public) | `CH_PORT` | -| Redis | `6379` | `127.0.0.1` (dev: public) | `REDIS_PORT` | +| Gateway | `8090` | `0.0.0.0` | `AGENTCC_GATEWAY_PORT` | +| Model serving | `8080` | `0.0.0.0` | `SERVING_PORT` | +| Code executor | `8060` | `0.0.0.0` | `CODE_EXECUTOR_PORT` | +| Postgres | `5432` | `127.0.0.1` | `PG_PORT` | +| ClickHouse HTTP | `8123` | `127.0.0.1` | `CH_HTTP_PORT` | +| ClickHouse TCP | `9000` | `127.0.0.1` | `CH_PORT` | +| Redis | `6379` | `127.0.0.1` | `REDIS_PORT` | | MinIO API | `9005` | `127.0.0.1` | `MINIO_API_PORT` | | MinIO console | `9006` | `127.0.0.1` | `MINIO_CONSOLE_PORT` | -| Temporal | `7233` | `127.0.0.1` (dev: public) | `TEMPORAL_PORT` | -| Temporal UI | `8085` | Dev mode only | `TEMPORAL_UI_PORT` | +| Temporal | `7233` | `127.0.0.1` | `TEMPORAL_PORT` | | PeerDB server | `9900` | `127.0.0.1` | `PEERDB_PORT` | | PeerDB UI | `3001` | `0.0.0.0` | `PEERDB_UI_PORT` | -In production, only the frontend and backend ports should be internet-facing, and only behind a TLS-terminating reverse proxy. +The data stores (Postgres, ClickHouse, Redis, MinIO, Temporal) bind to `127.0.0.1`; the application services bind to `0.0.0.0`. PeerDB server and UI only run when you enable the CDC stack with `COMPOSE_PROFILES=full`, so those two ports are only in use in that mode. -## Next Steps +## Dive deeper - - Clone, configure, and run the full stack. + + Clone the repo and bring the stack up with `./bin/install` - - Set secrets and tune runtime flags before first boot. + + Set provider keys, secrets, and runtime flags in `.env` diff --git a/src/pages/docs/self-hosting/support.mdx b/src/pages/docs/self-hosting/support.mdx new file mode 100644 index 00000000..9ba30624 --- /dev/null +++ b/src/pages/docs/self-hosting/support.mdx @@ -0,0 +1,32 @@ +--- +title: "Support" +description: "Where to get help running a self-hosted Future AGI instance" +--- + +Running the open-source stack and hit something these pages don't cover? Here's where to reach the team and the community, and what to include so you get a useful answer fast. + +## Where to get help + + + + Ask the community and the team in the Future AGI Discord + + + Report a bug or request a feature on the open-source repo + + + +## Before you post + +A self-hosting question is easier to answer with the basics attached: + +- What you ran and what happened, with the exact error +- Output of `docker compose ps` so the team can see which service is down +- Logs from the failing service: `docker compose logs --tail=100` +- Your platform (Linux host, EC2, GCE) and whether you're on managed data stores + +Most self-hosting questions are already answered in [Troubleshooting & FAQs](/docs/self-hosting/troubleshooting). Check there first. + +## Commercial support + +For managed hosting, an SLA, or help with a production rollout, reach out at [sales@futureagi.com](mailto:sales@futureagi.com). diff --git a/src/pages/docs/simulation/features/voice-replay.mdx b/src/pages/docs/simulation/features/voice-replay.mdx index 29403f52..6b75e1d4 100644 --- a/src/pages/docs/simulation/features/voice-replay.mdx +++ b/src/pages/docs/simulation/features/voice-replay.mdx @@ -33,7 +33,7 @@ The flow is: **select voice traces** → **create a replay session** → **gener - With **Voice Observability** integrated, your production voice calls (via Vapi, Retell, or other supported providers) are captured as traces with conversation-type spans. Each span stores the full call data including transcripts, recordings, and call metrics. See [Set Up Voice Observability](/docs/observe/features/voice) for integration details. + With **Voice Observability** integrated, your production voice calls (via Vapi, Retell, or other supported providers) are captured as traces with conversation-type spans. Each span stores the full call data including transcripts, recordings, and call metrics. See [Set Up Voice Observability](/docs/observe/concepts/voice-observability) for integration details. From the **Observe** experience, select the voice traces you want to replay. Create a **replay session** with: @@ -103,7 +103,7 @@ The **Compare with baseline call** button only appears for call executions that Replay text-based production sessions using chat simulation. - + Set up voice call monitoring for production calls. diff --git a/src/pages/docs/tracing/concepts/index.mdx b/src/pages/docs/tracing/concepts/index.mdx index 7250aed2..0b6c889a 100644 --- a/src/pages/docs/tracing/concepts/index.mdx +++ b/src/pages/docs/tracing/concepts/index.mdx @@ -19,7 +19,7 @@ Your app emits **spans** (LLM calls, tool calls, chain steps) via OpenTelemetry Your App → traceAI / OpenTelemetry SDK → OTLP (HTTP or gRPC) → Future AGI Backend → Observe Dashboard ``` -Each **trace** is one request or execution. Each **span** is one operation (LLM, tool, retriever, etc.) with input, output, timing, and optional cost and tokens. That data powers the entire UI: trace list, span detail, [sessions](/docs/observe/features/session), [evals](/docs/observe/features/evals), and [alerts](/docs/observe/features/alerts). +Each **trace** is one request or execution. Each **span** is one operation (LLM, tool, retriever, etc.) with input, output, timing, and optional cost and tokens. That data powers the entire UI: trace list, span detail, [sessions](/docs/observe/concepts/sessions), [evals](/docs/observe/guides/setup-evals), and [alerts](/docs/observe/guides/setup-alerts). --- diff --git a/src/pages/docs/tracing/concepts/spans.mdx b/src/pages/docs/tracing/concepts/spans.mdx index cd7699f6..f8482b2d 100644 --- a/src/pages/docs/tracing/concepts/spans.mdx +++ b/src/pages/docs/tracing/concepts/spans.mdx @@ -78,5 +78,5 @@ For example, if a span invokes an LLM, the model name, the invocation parameters Semantic Attributes are standardized naming conventions for common metadata present in typical operations. Using semantic attribute naming is recommended to ensure consistency across systems. -> See [semantic conventions](/docs/observe/features/manual-tracing/semantic-conventions) for more information. +> See [semantic conventions](/docs/sdk/tracing/semantic-conventions) for more information.