diff --git a/content/docs/user-guide/agent-observability/index.mdx b/content/docs/user-guide/agent-observability/index.mdx index 5a557c4..9c08ab9 100644 --- a/content/docs/user-guide/agent-observability/index.mdx +++ b/content/docs/user-guide/agent-observability/index.mdx @@ -5,17 +5,17 @@ import { IconDatabase, IconCode, IconTerminal2 } from '@tabler/icons-react'; -Agents are already becoming part of everyday software, sometimes users interact with them directly, and sometimes they run quietly behind a workflow. A single request can turn into a chain of model calls, tool calls, external APIs, retries, and intermediate outputs before the final answer is returned. +Agents are already becoming part of everyday software. Sometimes users work with them directly. Sometimes they sit behind a workflow and quietly make decisions on their own. In both cases, a single request can quickly turn into multiple model calls, tool calls, retries, and intermediate steps before the final answer shows up. -When that final answer is wrong, slow, or hard to explain, the response alone does not tell you enough. You need to see the complete invocation. This includes the original prompt, the model used, the tools called, the input sent to those tools, the output received, token usage, latency, and where the answer started to drift. That is the problem Agent Observability is meant to solve. +That is usually where debugging gets hard. When the final answer is wrong, slow, or incomplete, the answer itself is not enough. You need to see the full run: what prompt started it, which model was used, what tools were called, what each tool received, what came back, how many tokens were spent, and where the run started to drift. That is the job of Agent Observability. ## Agent observability with Parseable -Parseable Agent Observability gives you that view in Prism. You can start with a high-level view of agent activity, move into a single invocation, and then inspect the exact LLM call or tool execution that explains what happened. +Parseable Agent Observability gives you that view inside Prism. You can begin with a high-level view of agent activity, move into a single invocation, and then drill down to the exact LLM call or tool execution that explains what happened. -This is built on the [OpenTelemetry GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai), so telemetry follows a standard shape for model calls, token usage, inputs, outputs, tool calls, and agent spans. Instead of putting agent telemetry in a custom black box, Parseable lets you observe it alongside the rest of your telemetry. +It is built on the [OpenTelemetry GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai), so the telemetry follows a standard shape for model calls, token usage, inputs, outputs, tool calls, and agent spans. That means you are not pushing agent telemetry into a custom black box. You are sending it in a format that stays close to the OpenTelemetry model and remains easy to reason about. -Once your agent is instrumented, the Agents page becomes the place to understand how your agents are behaving in production. You can see how often they run, which models they use, how many tokens they consume, where they spend time, what they call, and where the answer starts to drift. +Once your agent is instrumented, the Agents page becomes the place where you understand how that agent behaves in production. You can see how often it runs, which models it uses, how many tokens it consumes, where it spends time, what it calls, and where the answer starts to go off track. Parseable enriches GenAI telemetry at ingest time with fields like `p_genai_tokens_total`, `p_genai_tokens_per_sec`, and `p_genai_duration_ms`, so token usage and latency analysis is available without extra client-side tracking. @@ -27,7 +27,7 @@ Parseable enriches GenAI telemetry at ingest time with fields like `p_genai_toke ![Agents overview showing invocations, errors, token usage, and duration charts](./static/agent-overview.png) -When you open the Agents page, you first land on the Overview tab. Start here when you want a quick health check of your agents over a selected time range. +When you open the Agents page, this is where you land first. The Overview tab is the quickest way to understand whether things look healthy over a selected time range. The Invocations chart shows how many agent runs happened and how many of them ended in error. Token usage is split into input tokens and output tokens, grouped by model, so you can quickly see which model is consuming the most tokens. The Duration section helps you compare latency by tool and by model. @@ -37,7 +37,7 @@ Use the time range picker in the top-right corner to change the window you are i ![Invocations tab showing session metrics, timeline, table columns, operations, and sentiment](./static/agent-invocations.png) -When you need to inspect actual runs, switch to the Invocations tab. This is where the aggregate view turns into individual agent sessions. +When you want to move from trends to actual runs, switch to the Invocations tab. This is where the broad view becomes a list of real agent sessions that you can inspect one by one. At the top, Parseable shows total sessions, total tokens, duration, and error rate for the selected period. Below that, the timeline shows when invocations happened and where errors appeared. @@ -49,7 +49,7 @@ You can also change the duration percentile from the dropdown in the Duration ca ![Invocations filter menu showing fields for tool, model, duration, tokens, and errors](./static/agent-filters.png) -When the list gets large, filters help you reduce the noise. Use Add filter to narrow the investigation to a smaller set of invocations. +Once the list grows, filters help you cut through the noise. Use Add filter when you already have a rough idea of what you are looking for and want to narrow the list to a smaller set of invocations. You can filter by fields such as tool name, tool call count, model name, invocation duration, input tokens, output tokens, total tokens, and total errors. This is useful when you already know the shape of the problem. For example, you can filter to a specific model, look at sessions with a high token count, or focus only on invocations where a tool failed. @@ -59,7 +59,7 @@ The Find in data search box helps you search within the visible invocation data ![Invocation detail showing status, sentiment timeline, waterfall tree, messages, and tool input output](./static/agent-invocation-detail.png) -The real debugging starts when you open a row from the Invocations table. This opens the invocation detail view for that specific agent run. +This is where the real investigation starts. Open any row from the Invocations table to see the full detail view for that run. The header shows the status, duration, tokens used, AI model, created time, and ended time. The Sentiment timeline shows how the interaction changed across the conversation, so you can see where the answer stayed useful, where satisfaction dipped, and where the run may need review. @@ -69,21 +69,20 @@ Use this view when you need to understand which prompt started the run, which mo ## Manual instrumentation using the OpenTelemetry SDK -You can manually create spans with `gen_ai.*` attributes using the OpenTelemetry SDK directly. -This lets you instrument any provider, framework, or custom LLM client, and add the exact attributes your team needs for debugging and analysis. +If you want to instrument your own agent, you can do that directly with the OpenTelemetry SDK. This works well when you have a custom agent, a provider-specific client, or a framework setup where you want full control over what gets captured. At minimum, instrument these parts of the agent workflow: 1. **Agent-level spans:** `invoke_agent` spans that wrap the full agent loop 2. **Chat spans:** one `chat {model}` span for every LLM request 3. **Tool execution spans:** one `execute_tool {tool.name}` span for every tool or command execution -4. **GenAI event logs:** user messages, assistant messages, choices, tool calls, tool inputs, tool outputs, and reasoning blocks +4. **Span events:** user messages, assistant messages, choices, tool calls, tool inputs, tool outputs, reasoning blocks, and evaluation results attached to the active span -Refer the [manual instrumentation guide](/docs/user-guide/agent-observability/manual-instrumentation) for detailed instructions on how to structure spans and what attributes to capture for complete agent observability. +The [manual instrumentation guide](/docs/user-guide/agent-observability/manual-instrumentation) shows how to structure those spans and what to capture so the data shows up cleanly in Parseable. ## Language support matrix -Any language that can emit OpenTelemetry traces and logs over OTLP can send agent observability data to Parseable. The manual guide currently uses Python examples, but the schema is language-agnostic. +Any language that can emit OpenTelemetry traces over OTLP can send agent observability data to Parseable. The manual guide currently uses Python examples, but the schema is language-agnostic. | Language | Manual OpenTelemetry SDK | |------------|-----------------| @@ -97,11 +96,11 @@ Any language that can emit OpenTelemetry traces and logs over OTLP can send agen ## Collector configuration -After the instrumentation is set up in your application, configure how telemetry data is sent to Parseable. We recommend using an OpenTelemetry Collector for buffering, batching, and reliability, but you can also export directly from your application for simpler setups. +After instrumentation is in place, the next step is deciding how you want to send the trace data to Parseable. For most teams, an OpenTelemetry Collector is the better default because it gives you batching and a more reliable path. If your setup is small, you can also export directly from the application. ### OpenTelemetry Collector (Recommended) -Deploy an OpenTelemetry Collector between your application and Parseable for buffering, batching, and reliability. Save the following as `parseable-genai-collector.yaml`: +Put an OpenTelemetry Collector between your application and Parseable when you want a cleaner and more production-friendly path for export. Save the following as `parseable-genai-collector.yaml`: ```yaml receivers: @@ -147,7 +146,7 @@ otelcol-contrib --config parseable-genai-collector.yaml ### Direct to Parseable -For simpler setups, you can export traces directly from your application to Parseable by setting OTLP environment variables. No collector process is needed. +For smaller setups, you can send traces directly from the application to Parseable with OTLP environment variables. This avoids running a separate collector process. ```bash export OTEL_EXPORTER_OTLP_ENDPOINT=${PARSEABLE_URL} @@ -161,7 +160,7 @@ export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true } title='Manual instrumentation guide'> -Complete reference for instrumenting any GenAI agent with OpenTelemetry traces and correlated logs. +Complete reference for instrumenting any GenAI agent with OpenTelemetry traces, spans, and events. } title='Schema Reference'> diff --git a/content/docs/user-guide/agent-observability/manual-instrumentation.mdx b/content/docs/user-guide/agent-observability/manual-instrumentation.mdx index 95bee74..5017f94 100644 --- a/content/docs/user-guide/agent-observability/manual-instrumentation.mdx +++ b/content/docs/user-guide/agent-observability/manual-instrumentation.mdx @@ -2,29 +2,30 @@ title: Manual instrumentation guide --- -This page is a reference manual on how to instrument your agent with OpenTelemetry spans and logs. Use it to capture the full agent run, every LLM call, every tool execution, and the GenAI event logs needed to reconstruct what happened. +This page shows how to instrument an agent in a way that makes sense once the data reaches Parseable. The goal is simple: when you open an invocation in the Agents view, you should be able to follow the run from start to finish without guessing what happened in between. + +The main thing to keep in mind is that agent observability here is trace-first. You do not need to think in terms of one pipeline for traces and another one for conversation logs. The agent run, each LLM call, and each tool execution are spans. The prompt, response, tool input, tool output, reasoning details, and evaluation results live as events attached to the relevant span inside that same trace. + +Parseable natively supports the OpenTelemetry GenAI semantic conventions, so if you already emit standard `gen_ai.*` fields, they stay close to the OpenTelemetry model and remain easy to query later. ## Instrumentation overview -For each agent run, you want to capture: +For each agent run, capture these pieces. If you get these four parts right, the rest of the debugging experience becomes much easier. | What | Where it goes | Why | |------|--------------|-----| -| **Agent run:** which agent ran, total tokens, exit status | OTel **`invoke-agent` span** | End-to-end agent observability, run-level dashboards | -| **LLM call metadata:** model name, token counts, latency, temperature, finish reason, errors | OTel **`chat` span** | Per-call dashboards, aggregation, alerting, usage tracking | -| **Full conversation content:** system prompts, user messages, assistant responses, tool results | OTel **log records** (correlated to `chat` spans) | Conversation reconstruction, debugging, quality analysis | -| **Tool calls:** which tools the LLM called, with full arguments | OTel **log records** (correlated to `chat` spans) + **`execute_tool` spans** | Tool usage analytics, debugging | -| **Thinking/reasoning:** Claude's chain-of-thought reasoning blocks | OTel **log records** (correlated to `chat` spans) | Reasoning analysis, debugging | - -The OpenTelemetry GenAI semantic conventions define how to structure this data. This guide shows exactly how to implement it. +| **Agent run:** which agent ran, total tokens, exit status | OTel **`invoke-agent` span** | End-to-end agent observability and run-level summaries | +| **LLM call metadata:** model, token counts, latency, finish reason, errors | OTel **`chat` span** | Per-call analysis, latency review, usage tracking | +| **Tool execution:** which tool ran, how long it took, whether it failed | OTel **`execute_tool` span** | Tool debugging and operation-level visibility | +| **Prompt, response, tool I/O, reasoning, evaluation results** | OpenTelemetry **events** attached to the active span | Conversation reconstruction, tool debugging, and answer quality review | ## Manual instrumentation rules -Keep manual instrumentation intentionally small. Instrument only agent runs, LLM calls, tool executions, and their event logs. +Keep manual instrumentation small and predictable. You do not need to wrap every helper function in telemetry. Focus on the parts that explain how the agent made progress, where it spent time, and where it failed. ### Agent invocation -Create one parent span for the full agent invocation. +Start with one parent span for the full agent invocation. This is the span that ties the whole run together. - Span name: `invoke-agent` - Required span attributes: @@ -33,18 +34,20 @@ Create one parent span for the full agent invocation. - `gen_ai.provider.name` - `gen_ai.request.model` - Completion span attributes: - - `gen_ai.usage.input_tokens`: total input tokens across all chat calls - - `gen_ai.usage.output_tokens`: total output tokens across all chat calls + - `gen_ai.usage.input_tokens` + - `gen_ai.usage.output_tokens` - `gen_ai.response.finish_reasons` -- Required logs: - - `gen_ai.user.message`: task, prompt, issue, or problem statement that started the run - - `gen_ai.agent.finish`: final status, total tokens, submission/result metadata -All child `chat {model}` and `execute_tool {tool.name}` spans should be created inside this span context so the trace waterfall shows the full agent decision path. +At this level, the most useful events are: + +- `gen_ai.user.message`: the task, prompt, issue, or problem statement that started the run +- `gen_ai.agent.finish`: completion summary such as exit status or submission state + +Every `chat {model}` span and every `execute_tool {tool.name}` span should be created inside this parent context. That is what gives you a clean waterfall later in the UI. ### Chat calls -Create one child span for each LLM request. +Create one child span for each LLM request. If the agent calls the model three times, you should see three chat spans. - Span name: `chat {model}` - Required span attributes: @@ -60,23 +63,24 @@ Create one child span for each LLM request. - `gen_ai.usage.input_tokens` - `gen_ai.usage.output_tokens` -Attach input and output tokens to every successful chat span. Prefer provider usage fields, for example `response.usage.prompt_tokens` and `response.usage.completion_tokens`. Do not compute output tokens only from assistant message text: tool-call responses often have empty `message.content` even though `completion_tokens` is non-zero. - -Emit logs inside the active chat span for: +Inside that chat span, add events for the parts of the interaction you may want to inspect later: +- `gen_ai.client.inference.operation.details` when your SDK supports the OpenTelemetry GenAI event shape - `gen_ai.system.message` - `gen_ai.user.message` - `gen_ai.assistant.message` - `gen_ai.tool.message` - `gen_ai.choice` - `gen_ai.tool.call` -- `gen_ai.thinking` when the provider exposes reasoning or thinking blocks +- `gen_ai.thinking` when the provider exposes reasoning blocks + +Attach input and output token counts to every successful chat span. Prefer the usage values returned by the provider rather than trying to estimate them yourself. -Each log should include relevant `gen_ai.*` fields as log attributes, not only as span attributes. This keeps logs queryable without joining back to traces. +The OpenTelemetry GenAI conventions define `gen_ai.client.inference.operation.details` for the full request and response details of a model call. Use it when you can, especially when you want to keep `gen_ai.input.messages`, `gen_ai.output.messages`, system instructions, tool definitions, and request settings together in one structured event. If your SDK does not support that event shape yet, use the per-message events above and keep their names stable. ### Tool execution -Create one child span for each tool execution. +Create one child span for each tool execution. This keeps the LLM request separate from the tool that actually ran because those are usually two different parts of the story. - Span name: `execute_tool {tool.name}` - Required span attributes: @@ -85,61 +89,164 @@ Create one child span for each tool execution. - `gen_ai.tool.type` - `gen_ai.tool.call.id` when available -Emit logs inside the active tool span for: +Inside the tool span, add events for: - `gen_ai.tool.input`: command, function arguments, request body, or tool input - `gen_ai.tool.output`: observation, result, response body, or failure output -Use `gen_ai.tool.call.id` on both chat and tool telemetry when function calling provides an ID. This lets you correlate `gen_ai.tool.call` logs from a chat span with the matching `execute_tool {tool.name}` span. +When the model provides a tool call ID, use the same `gen_ai.tool.call.id` on both the chat span and the tool span. That makes it much easier to connect the model request with the real execution step. + +### Events + +Events are the detail layer of the trace. Spans tell you that an agent called a model or executed a tool. Events tell you what the model saw, what it returned, which tool it requested, what input the tool received, what output came back, and how the answer was evaluated. + +Use events for content you may need to read during an investigation. Keep span attributes for fields you want to group, filter, and aggregate often. + +For most agents, these event groups are enough: + +| Event group | Use it for | Attach it to | +|-------------|------------|--------------| +| `gen_ai.client.inference.operation.details` | Full model request and response details when supported by your SDK | `chat {model}` span | +| Message events such as `gen_ai.user.message` and `gen_ai.assistant.message` | Individual system, user, assistant, and tool messages | `chat {model}` span | +| `gen_ai.choice` | The model output choice, finish reason, and returned content | `chat {model}` span | +| `gen_ai.tool.call` | The tool call requested by the model | `chat {model}` span | +| `gen_ai.tool.input` and `gen_ai.tool.output` | The actual tool arguments and result | `execute_tool {tool.name}` span | +| `gen_ai.thinking` | Provider-specific reasoning or thinking blocks when exposed | `chat {model}` span | +| `gen_ai.evaluation.result` | Answer quality, sentiment, correctness, relevance, or other evaluator output | Span being evaluated, usually `chat {model}` or `invoke-agent` | + +For evaluation events, include fields such as `gen_ai.evaluation.name`, `gen_ai.evaluation.score.label`, `gen_ai.evaluation.score.value`, and `gen_ai.evaluation.explanation` when you have them. This helps the invocation view explain not only what the agent did, but also whether the answer looked useful after the run. + +If your application uses an event helper such as `_otel.emit(...)`, keep the shape consistent: + +```python +import json + + +def emit_genai_event(span, event_name: str, body, attributes: dict | None = None): + event_attributes = { + **(attributes or {}), + "gen_ai.event.name": event_name, + } + + if isinstance(body, (dict, list)): + event_attributes["body"] = json.dumps(body) + else: + event_attributes["body"] = body + + span.add_event(event_name, event_attributes) +``` + +Use the event name both as the OpenTelemetry event name and as `gen_ai.event.name`. Parseable keeps the event name in `event_name` and prefixes event attributes with `event_`, so `gen_ai.event.name` becomes `event_gen_ai.event.name` and `body` becomes `event_body`. + +For example: + +```python +emit_genai_event( + span, + "gen_ai.user.message", + body=problem_statement, + attributes={ + **invoke_attrs, + "role": "user", + }, +) + +emit_genai_event( + span, + "gen_ai.agent.finish", + body={ + "exit_status": exit_status or "stop", + "total_input_tokens": total_input_tokens, + "total_output_tokens": total_output_tokens, + }, + attributes={ + **invoke_attrs, + "gen_ai.usage.input_tokens": total_input_tokens, + "gen_ai.usage.output_tokens": total_output_tokens, + }, +) + +emit_genai_event( + tool_span, + "gen_ai.tool.input", + body=tool_input, + attributes=tool_attrs, +) + +emit_genai_event( + tool_span, + "gen_ai.tool.output", + body=tool_output, + attributes={ + **tool_attrs, + "swe_agent.tool.output_bytes": len(tool_output.encode(errors="replace")), + "swe_agent.tool.execution_time.seconds": execution_time, + }, +) + +emit_genai_event( + chat_span, + "gen_ai.tool.call", + body=tool_call.get("function", tool_call), + attributes=tool_call_attrs, +) + +emit_genai_event( + chat_span, + "gen_ai.thinking", + body=thinking_text, + attributes=response_attrs, +) +``` ### Errors -Failures should mark the active span as errored. +When something fails, mark the active span as failed. This is what makes bad runs easy to spot in the invocation view. For chat failures: -- Mark the `chat {model}` span with error status. -- Set `error.type` to the exception class, provider error class, or provider error code. -- Use the error message as the status description. -- Emit any available provider metadata as `gen_ai.*` attributes. +- mark the `chat {model}` span with error status +- set `error.type` +- use the error message as the status description when available For tool failures: -- Mark the `execute_tool {tool.name}` span with error status. -- Set `error.type`. -- Emit a `gen_ai.tool.output` log containing the failure output or error string. +- mark the `execute_tool {tool.name}` span with error status +- set `error.type` +- add a `gen_ai.tool.output` span event that carries the failure output or error string ### Field placement -Expose every relevant `gen_ai.*` field in both traces and logs: +As a rule of thumb, put the fields you want to aggregate often on spans. Put the full content you want to inspect later on events. + +That usually means: + +- spans hold model, provider, latency, token counts, status, and tool name +- events hold prompts, responses, tool input, tool output, reasoning blocks, and evaluation results -- Put run, chat, tool, token, model, provider, response, and error fields on spans. -- Put `gen_ai.event.name` and the same query-critical `gen_ai.*` fields on logs. -- Keep full message/tool/reasoning content in the log body. -- Avoid truncating content in instrumentation unless your security or retention policy requires it. +This keeps the full run in one trace while still giving Parseable enough detail to show both the summary and the step-by-step interaction. ## Architecture ``` ┌─────────────────────────────────────────────────────────┐ -│ Your Agent Application (Python) │ +│ Your Agent Application │ │ │ │ ┌──────────────────────────────────────────────────┐ │ │ │ opentelemetry-instrument CLI wrapper │ │ -│ │ (configures TracerProvider, LoggerProvider, │ │ -│ │ and OTLP exporters) │ │ +│ │ (configures TracerProvider and OTLP exporter) │ │ │ └──────────────────────────────────────────────────┘ │ │ │ │ ┌─────────────────────────────────────────────────┐ │ │ │ Your code (manual instrumentation): │ │ -│ │ - _tracer.start_as_current_span(...) → spans │ │ -│ │ - _otel_logger.emit(...) → logs │ │ +│ │ - start_as_current_span(...) → spans │ │ +│ │ - span.add_event(...) → span events │ │ │ └─────────────────────────────────────────────────┘ │ │ │ │ │ │ OTLP (protobuf) │ │ ▼ │ │ ┌─────────────────────────────────────────────────┐ │ -│ │ BatchSpanProcessor + BatchLogRecordProcessor │ │ +│ │ BatchSpanProcessor │ │ │ │ (buffers, batches, exports periodically) │ │ │ └─────────────────────────────────────────────────┘ │ └────────────────────│────────────────────────────────────┘ @@ -147,17 +254,16 @@ Expose every relevant `gen_ai.*` field in both traces and logs: ▼ ┌─────────────────────────────────────────────────────────┐ │ OTel Collector │ -│ - Receives traces + logs via OTLP │ +│ - Receives traces via OTLP │ │ - Batches and exports to backend(s) │ -│ - Routes traces and logs to separate streams │ └────────────────────│────────────────────────────────────┘ │ OTLP/HTTP (JSON) ▼ ┌─────────────────────────────────────────────────────────┐ │ Parseable │ -│ - genai-traces stream (flattened spans) │ -│ - genai-logs stream (flattened log records) │ -│ - SQL-queryable, with token and duration enrichment │ +│ - receives trace spans and span events │ +│ - flattens both into SQL-queryable records │ +│ - enriches token and duration fields at ingest │ └─────────────────────────────────────────────────────────┘ ``` @@ -165,63 +271,13 @@ Expose every relevant `gen_ai.*` field in both traces and logs: | Layer | Responsibility | |-------|----------------| -| `opentelemetry-instrument` CLI | Bootstraps the SDK, creates `TracerProvider` and `LoggerProvider`, and configures OTLP exporters. You do not call `set_tracer_provider()` or `set_logger_provider()` manually. | -| Your code (manual instrumentation) | Creates spans (`_tracer.start_as_current_span(...)`) and emits log records (`_otel_logger.emit(...)`). This is where all GenAI-specific attributes and content are set. | -| `BatchSpanProcessor` / `BatchLogRecordProcessor` | Accumulates spans and logs in memory, exports them in batches via OTLP. Configured by the CLI. | -| OTel Collector | Receives OTLP data, applies processors (batching, filtering), and exports to one or more backends. Converts protobuf to JSON for Parseable. | -| Parseable | Stores traces and logs as flattened, SQL-queryable records. Enriches with computed columns such as `p_genai_tokens_total`, `p_genai_tokens_per_sec`, and `p_genai_duration_ms`. | - -## The opentelemetry-instrument CLI - -The `opentelemetry-instrument` CLI is the simplest way to bootstrap the OTel SDK. It is installed as part of the `opentelemetry-distro` package and is the recommended approach for Python applications. - -### What It Does - -When you run `opentelemetry-instrument python my_agent.py`, the CLI: - -1. Creates a `TracerProvider` with a `BatchSpanProcessor` and OTLP exporter -2. Creates a `LoggerProvider` with a `BatchLogRecordProcessor` and OTLP exporter -3. Sets both as the global providers (so `trace.get_tracer()` and `get_logger_provider()` return them) -4. Runs your application with the configured OpenTelemetry providers - -### What You Do NOT Do +| `opentelemetry-instrument` CLI | Bootstraps the SDK, creates a `TracerProvider`, and configures the OTLP exporter. | +| Your code | Creates spans and span events. This is where GenAI attributes and event content are set. | +| `BatchSpanProcessor` | Buffers spans in memory and exports them in batches via OTLP. | +| OTel Collector | Receives OTLP traces, applies processors such as batching, and exports them onward. | +| Parseable | Stores span rows and event rows as flattened trace records with GenAI enrichments such as `p_genai_tokens_total`, `p_genai_tokens_per_sec`, and `p_genai_duration_ms`. | -Because the CLI handles provider setup: - -- Do NOT call `set_tracer_provider()` because it is already set -- Do NOT call `set_logger_provider()` because it is already set -- Do NOT create `BatchSpanProcessor` or `OTLPSpanExporter` because they are already configured -- Do NOT create `BatchLogRecordProcessor` or `OTLPLogExporter` because they are already configured -- Just call `trace.get_tracer(...)` and `get_logger_provider().get_logger(...)` because they return the pre-configured providers - -### Manual Provider Setup (Without CLI) - -If you cannot use the CLI (e.g., embedded in a larger application), you can set up providers manually: - -```python -from opentelemetry import trace -from opentelemetry.sdk.trace import TracerProvider -from opentelemetry.sdk.trace.export import BatchSpanProcessor -from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter -from opentelemetry.sdk._logs import LoggerProvider -from opentelemetry.sdk._logs.export import BatchLogRecordProcessor -from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter -from opentelemetry._logs import set_logger_provider - -# Traces -tracer_provider = TracerProvider() -tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) -trace.set_tracer_provider(tracer_provider) - -# Logs -logger_provider = LoggerProvider() -logger_provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter())) -set_logger_provider(logger_provider) -``` - -This is the manual equivalent of what the CLI does. Use the CLI when possible. - -## Setup: Packages, Environment, Launch +## Setup: packages, environment, launch ### Packages @@ -232,260 +288,177 @@ pip install opentelemetry-distro opentelemetry-exporter-otlp | Package | Purpose | |---------|---------| | `opentelemetry-distro` | Provides the `opentelemetry-instrument` CLI | -| `opentelemetry-exporter-otlp` | OTLP exporter (HTTP/protobuf and gRPC) | +| `opentelemetry-exporter-otlp` | OTLP exporter for traces | -### Environment Variables +### Environment variables ```bash -# Required: where the OTel Collector listens export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 - -# Required: OTLP protocol export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf - -# Required: identifies your application in traces export OTEL_SERVICE_NAME=my-genai-agent - -# Optional: include full GenAI message content when your instrumentation emits it export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true ``` | Variable | Purpose | Required | |----------|---------|----------| | `OTEL_EXPORTER_OTLP_ENDPOINT` | Collector address. `4318` for HTTP, `4317` for gRPC. | Yes | -| `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` (HTTP) or `grpc` | Yes | -| `OTEL_SERVICE_NAME` | Service name that appears on every span and log record as a resource attribute | Yes | -| `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` | Keeps GenAI message content available when your instrumentation emits it. Review your privacy policy before enabling this in production. | No | +| `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` or `grpc` | Yes | +| `OTEL_SERVICE_NAME` | Service name that appears on every span | Yes | +| `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` | Keeps GenAI message content available when your instrumentation emits it as span events. Review privacy policy before enabling this in production. | No | -### Launch Command +### Launch command ```bash opentelemetry-instrument \ --traces_exporter otlp \ - --logs_exporter otlp \ + --logs_exporter none \ --metrics_exporter none \ python my_agent.py ``` -| Flag | Value | Purpose | -|------|-------|---------| -| `--traces_exporter` | `otlp` | Export spans via OTLP | -| `--logs_exporter` | `otlp` | Export log records via OTLP | -| `--metrics_exporter` | `none` | Disable metrics. Optional, set to `otlp` if you want metrics. | +The CLI reads `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_EXPORTER_OTLP_PROTOCOL` to configure the exporter. -The CLI reads `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_EXPORTER_OTLP_PROTOCOL` to configure the exporters. +## Module-level setup -## Instrumentation Code - -This is a complete, agent-agnostic reference for instrumenting any Python application that calls LLM APIs. The instrumentation uses three span types per the [OTel GenAI Agent Spans](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/) spec. - -### Module-Level Setup - -You need a tracer and logger in the agent orchestration module (for `invoke_agent` and `execute_tool` spans + logs), and a tracer and logger in the LLM call module (for `chat` spans and log records): - -**Agent orchestration module** (e.g., `agents.py`): +You only need a tracer: ```python from opentelemetry import trace -from opentelemetry._logs import get_logger_provider, SeverityNumber _tracer = trace.get_tracer("my-agent", "1.1.0") -_otel_logger = get_logger_provider().get_logger("my-agent", "1.1.0") ``` -**LLM call module** (e.g., `models.py`): +Use a dotted scope name when you want to separate orchestration and model-call code, for example `my-agent` and `my-agent.llm`. + +## Instrumenting the agent run + +Wrap the full agent loop in an `invoke-agent` span. ```python import json from opentelemetry import trace -from opentelemetry._logs import get_logger_provider, SeverityNumber -_tracer = trace.get_tracer("my-agent.llm", "1.1.0") -_otel_logger = get_logger_provider().get_logger("my-agent.llm", "1.1.0") -``` - -- The first argument is the **instrumentation scope name**. Use a dotted name that identifies which part of your application is emitting telemetry. Both spans and logs will carry this as `scope_name`. -- The second argument is the **instrumentation scope version**. Bump this when you change what attributes/events you emit. -- `get_logger_provider()` returns the provider already configured by the CLI. Do NOT create your own. - -### Instrumenting the Agent Run (invoke_agent) -Wrap the entire agent loop in an agent invocation span named `invoke-agent`. Set `gen_ai.operation.name` to `invoke_agent`. All `chat` and `execute_tool` spans created inside this context automatically become children via OTel context propagation. - -```python def run_agent(model_name: str, provider: str, problem: str): - """Example: wrap an agent run with an invoke_agent span.""" + total_input_tokens = 0 + total_output_tokens = 0 - span_name = "invoke-agent" - with _tracer.start_as_current_span(span_name, kind=trace.SpanKind.CLIENT) as span: + with _tracer.start_as_current_span("invoke-agent", kind=trace.SpanKind.CLIENT) as span: span.set_attribute("gen_ai.operation.name", "invoke_agent") span.set_attribute("gen_ai.agent.name", "my-agent") span.set_attribute("gen_ai.provider.name", provider) span.set_attribute("gen_ai.request.model", model_name) - # ── Emit the problem statement that triggered this agent run ── - _otel_logger.emit( - body=problem, - severity_number=SeverityNumber.INFO, - event_name="gen_ai.user.message", - attributes={ - "gen_ai.operation.name": "invoke_agent", - "gen_ai.provider.name": provider, - "gen_ai.request.model": model_name, - "gen_ai.agent.name": "my-agent", - "gen_ai.event.name": "gen_ai.user.message", + span.add_event( + "gen_ai.user.message", + { "role": "user", + "content": problem, }, ) - # ── Your agent loop ── done = False while not done: - llm_response = call_llm(model_name, messages) # creates a child "chat" span - tool_result = execute_tool(llm_response.action) # creates a child "execute_tool" span + llm_response = call_llm(model_name, provider, messages) + if llm_response.tool_call: + execute_tool(llm_response.tool_call) done = llm_response.is_done - # ── After loop: set aggregate response attributes ── span.set_attribute("gen_ai.usage.input_tokens", total_input_tokens) span.set_attribute("gen_ai.usage.output_tokens", total_output_tokens) span.set_attribute("gen_ai.response.finish_reasons", json.dumps(["exit_command"])) - # ── Emit agent completion summary ── - _otel_logger.emit( - body=json.dumps({"exit_status": "exit_command", - "total_input_tokens": total_input_tokens, - "total_output_tokens": total_output_tokens}), - severity_number=SeverityNumber.INFO, - event_name="gen_ai.agent.finish", - attributes={ - "gen_ai.operation.name": "invoke_agent", - "gen_ai.agent.name": "my-agent", - "gen_ai.provider.name": provider, - "gen_ai.request.model": model_name, - "gen_ai.event.name": "gen_ai.agent.finish", - "gen_ai.usage.input_tokens": total_input_tokens, - "gen_ai.usage.output_tokens": total_output_tokens, + span.add_event( + "gen_ai.agent.finish", + { + "exit_status": "exit_command", + "total_input_tokens": total_input_tokens, + "total_output_tokens": total_output_tokens, }, ) ``` -**Key points:** -- `kind=trace.SpanKind.CLIENT`: the agent is a client of the LLM service -- Token counts on `invoke_agent` are **totals** across all LLM calls in the run -- `gen_ai.agent.name` lives here, NOT on individual `chat` spans -- `gen_ai.provider.name` replaces the older `gen_ai.system` attribute per current spec -- The `gen_ai.user.message` log emits the problem statement so it appears in the trace waterfall -- The `gen_ai.agent.finish` log emits a completion summary with exit status and total token counts - -### Instrumenting Tool Execution (execute_tool) +## Instrumenting tool execution -Wrap each tool/command execution in an `execute_tool` span: +Wrap each tool execution in an `execute_tool` span. ```python -def execute_tool(action: str, tool_call_id: str | None = None): - """Example: wrap a tool execution with an execute_tool span.""" +from opentelemetry import trace + +def execute_tool(action: str, tool_call_id: str | None = None): tool_name = action.strip().split()[0] if action.strip() else "unknown" - span_name = f"execute_tool {tool_name}" - with _tracer.start_as_current_span(span_name, kind=trace.SpanKind.INTERNAL) as span: + + with _tracer.start_as_current_span( + f"execute_tool {tool_name}", + kind=trace.SpanKind.INTERNAL, + ) as span: span.set_attribute("gen_ai.operation.name", "execute_tool") span.set_attribute("gen_ai.tool.name", tool_name) span.set_attribute("gen_ai.tool.type", "function") if tool_call_id: span.set_attribute("gen_ai.tool.call.id", tool_call_id) - # Emit tool input log (the command being run) - _otel_logger.emit( - body=action, - severity_number=SeverityNumber.INFO, - event_name="gen_ai.tool.input", - attributes={ - "gen_ai.operation.name": "execute_tool", - "gen_ai.tool.name": tool_name, - "gen_ai.tool.type": "function", - "gen_ai.tool.call.id": tool_call_id or "", - "gen_ai.event.name": "gen_ai.tool.input", + span.add_event( + "gen_ai.tool.input", + { + "content": action, }, ) try: result = env.communicate(action) - except TimeoutError as e: - span.set_attribute("error.type", type(e).__name__) - span.set_status(trace.StatusCode.ERROR, "Command timed out") - _otel_logger.emit( - body=str(e), - severity_number=SeverityNumber.INFO, - event_name="gen_ai.tool.output", - attributes={ - "gen_ai.operation.name": "execute_tool", - "gen_ai.tool.name": tool_name, - "gen_ai.tool.type": "function", - "gen_ai.tool.call.id": tool_call_id or "", - "gen_ai.event.name": "gen_ai.tool.output", + except TimeoutError as exc: + span.set_attribute("error.type", type(exc).__name__) + span.record_exception(exc) + span.set_status(trace.Status(trace.StatusCode.ERROR, str(exc))) + span.add_event( + "gen_ai.tool.output", + { + "content": str(exc), }, ) raise - # Emit tool output log (the observation) - _otel_logger.emit( - body=result or "", - severity_number=SeverityNumber.INFO, - event_name="gen_ai.tool.output", - attributes={ - "gen_ai.operation.name": "execute_tool", - "gen_ai.tool.name": tool_name, - "gen_ai.tool.type": "function", - "gen_ai.tool.call.id": tool_call_id or "", - "gen_ai.event.name": "gen_ai.tool.output", + span.add_event( + "gen_ai.tool.output", + { + "content": result or "", }, ) return result ``` -**Key points:** -- `kind=trace.SpanKind.INTERNAL`: tool execution is an internal operation -- `gen_ai.tool.call.id` links this execution back to the LLM's tool call request (from function calling mode) -- Error status is set on timeout or failure, making failed tool executions queryable -- `gen_ai.tool.input` log captures the command sent to the tool -- `gen_ai.tool.output` log captures the tool's observation/result - -### Instrumenting an LLM Call (chat) +## Instrumenting an LLM call -Wrap your LLM call in a span. Emit log records inside the span context. +Wrap each LLM request in a `chat {model}` span and add the prompt, response, tool call, or reasoning details as span events. ```python -def call_llm(model: str, messages: list[dict], temperature: float = 0.0, **kwargs): - """Example: instrument any LLM call with OTel traces + logs.""" +import json +from opentelemetry import trace - with _tracer.start_as_current_span(f"chat {model}", kind=trace.SpanKind.CLIENT) as span: - # ── Step 1: Set request attributes on span ── +def call_llm(model: str, provider: str, messages: list[dict], temperature: float = 0.0, **kwargs): + with _tracer.start_as_current_span(f"chat {model}", kind=trace.SpanKind.CLIENT) as span: span.set_attribute("gen_ai.operation.name", "chat") + span.set_attribute("gen_ai.provider.name", provider) span.set_attribute("gen_ai.request.model", model) - span.set_attribute("gen_ai.provider.name", "openai") # or "anthropic", etc. if temperature is not None: span.set_attribute("gen_ai.request.temperature", temperature) - # ── Step 2: Emit input message log records ── for msg in messages: role = msg.get("role", "user") content = msg.get("content", "") body = content if isinstance(content, str) else json.dumps(content) - _otel_logger.emit( - body=body, - severity_number=SeverityNumber.INFO, - event_name=f"gen_ai.{role}.message", - attributes={ - "gen_ai.provider.name": "openai", - "gen_ai.request.model": model, - "gen_ai.event.name": f"gen_ai.{role}.message", + span.add_event( + f"gen_ai.{role}.message", + { "role": role, + "content": body, }, ) - # ── Step 3: Call the LLM ── try: response = your_llm_client.chat.completions.create( model=model, @@ -493,12 +466,12 @@ def call_llm(model: str, messages: list[dict], temperature: float = 0.0, **kwarg temperature=temperature, **kwargs, ) - except Exception as e: - span.set_status(trace.StatusCode.ERROR, str(e)) - span.set_attribute("error.type", type(e).__name__) + except Exception as exc: + span.set_attribute("error.type", type(exc).__name__) + span.record_exception(exc) + span.set_status(trace.Status(trace.StatusCode.ERROR, str(exc))) raise - # ── Step 4: Set response attributes on span ── if response.model: span.set_attribute("gen_ai.response.model", response.model) if response.id: @@ -507,172 +480,81 @@ def call_llm(model: str, messages: list[dict], temperature: float = 0.0, **kwarg span.set_attribute("gen_ai.usage.input_tokens", response.usage.prompt_tokens) span.set_attribute("gen_ai.usage.output_tokens", response.usage.completion_tokens) - # Prefer provider usage fields for token counts. Tool-call responses can - # have empty message.content even though completion_tokens is non-zero. - - # ── Step 5: Emit response log records ── finish_reasons = [] for i, choice in enumerate(response.choices): - # Choice content - _otel_logger.emit( - body=choice.message.content or "", - severity_number=SeverityNumber.INFO, - event_name="gen_ai.choice", - attributes={ - "gen_ai.provider.name": "openai", - "gen_ai.request.model": model, - "gen_ai.event.name": "gen_ai.choice", + span.add_event( + "gen_ai.choice", + { + "content": choice.message.content or "", "index": i, "finish_reason": choice.finish_reason or "", }, ) + if choice.finish_reason: finish_reasons.append(choice.finish_reason) - # Tool calls (if present) if choice.message.tool_calls: tool_call_ids = [] for tc in choice.message.tool_calls: tc_dict = tc.model_dump() if hasattr(tc, "model_dump") else tc - if tc_dict.get("id"): - tool_call_ids.append(tc_dict["id"]) - _otel_logger.emit( - body=json.dumps(tc_dict.get("function", tc_dict)), - severity_number=SeverityNumber.INFO, - event_name="gen_ai.tool.call", - attributes={ - "gen_ai.provider.name": "openai", - "gen_ai.request.model": model, - "gen_ai.event.name": "gen_ai.tool.call", + call_id = tc_dict.get("id", "") + if call_id: + tool_call_ids.append(call_id) + + span.add_event( + "gen_ai.tool.call", + { "gen_ai.tool.name": tc_dict.get("function", {}).get("name", ""), - "gen_ai.tool.call.id": tc_dict.get("id", ""), + "gen_ai.tool.call.id": call_id, + "content": json.dumps(tc_dict.get("function", tc_dict)), }, ) - # Set tool call IDs on the chat span for cross-span correlation - # with execute_tool spans. Single ID stored as string, multiple - # IDs JSON-encoded as array. if tool_call_ids: span.set_attribute( "gen_ai.tool.call.id", tool_call_ids[0] if len(tool_call_ids) == 1 else json.dumps(tool_call_ids), ) - # Thinking/reasoning blocks (Claude, DeepSeek, etc.) thinking_blocks = getattr(choice.message, "thinking_blocks", None) if thinking_blocks: for tb in thinking_blocks: thinking_text = tb.get("thinking", "") if isinstance(tb, dict) else str(tb) - _otel_logger.emit( - body=thinking_text, - severity_number=SeverityNumber.INFO, - event_name="gen_ai.thinking", - attributes={ - "gen_ai.provider.name": "openai", - "gen_ai.request.model": model, - "gen_ai.event.name": "gen_ai.thinking", + span.add_event( + "gen_ai.thinking", + { + "content": thinking_text, }, ) - # ── Step 6: Finalize span ── if finish_reasons: span.set_attribute("gen_ai.response.finish_reasons", json.dumps(finish_reasons)) - span.set_status(trace.StatusCode.OK) + span.set_status(trace.Status(trace.StatusCode.OK)) return response ``` -### What Gets Produced Per Agent Run - -**Three span types** (in the traces pipeline): - -**`invoke-agent` span:** one per agent run: - -| Attribute | Example | -|-----------|---------| -| `span_name` | `"invoke-agent"` | -| `gen_ai.operation.name` | `"invoke_agent"` | -| `gen_ai.agent.name` | `"my-agent"` | -| `gen_ai.provider.name` | `"openai"` | -| `gen_ai.request.model` | `"gpt-4o"` | -| `gen_ai.usage.input_tokens` | `45200` (total across all steps) | -| `gen_ai.usage.output_tokens` | `3800` (total across all steps) | -| `gen_ai.response.finish_reasons` | `["exit_command"]` | -| `span_kind` | `CLIENT` | -| `scope_name` | `"my-agent"` | -| `service.name` | `"my-genai-agent"` | - -**`chat` span:** one per LLM call, child of `invoke_agent`: - -| Attribute | Example | -|-----------|---------| -| `span_name` | `"chat gpt-4o"` | -| `gen_ai.operation.name` | `"chat"` | -| `gen_ai.request.model` | `"gpt-4o"` | -| `gen_ai.provider.name` | `"openai"` | -| `gen_ai.response.model` | `"gpt-4o-2024-11-20"` | -| `gen_ai.response.id` | `"chatcmpl-AZk8j..."` | -| `gen_ai.usage.input_tokens` | `1250` | -| `gen_ai.usage.output_tokens` | `380` | -| `gen_ai.request.temperature` | `0.0` | -| `gen_ai.response.finish_reasons` | `["stop"]` | -| `gen_ai.tool.call.id` | `"call_abc123"` (single) or `["call_abc123","call_def456"]` (multiple, JSON array) | -| `span_kind` | `CLIENT` | -| `span_status` | `OK` or `ERROR` | -| `error.type` | `"RateLimitError"` (only on error) | -| `scope_name` | `"my-agent.llm"` | - -**`execute_tool` span:** one per tool execution, child of `invoke_agent`: - -| Attribute | Example | -|-----------|---------| -| `span_name` | `"execute_tool find_file"` | -| `gen_ai.operation.name` | `"execute_tool"` | -| `gen_ai.tool.name` | `"find_file"` | -| `gen_ai.tool.type` | `"function"` | -| `gen_ai.tool.call.id` | `"call_abc123"` (when using function calling) | -| `span_kind` | `INTERNAL` | -| `span_status` | `OK` or `ERROR` | -| `error.type` | `"CommandTimeoutError"` (only on error) | -| `scope_name` | `"my-agent"` | - - -The `gen_ai.tool.call.id` attribute on `chat` spans enables cross-span correlation. You can JOIN a `chat` span to its corresponding `execute_tool` spans via the shared tool call ID. - - -**Multiple log records** (in the logs pipeline), carrying matching `trace_id` + `span_id` from their respective spans. Log records are emitted in all three span types: - -**`invoke_agent` log records:** - -| `event_name` | `body` content | When | -|-------------|---------------|------| -| `gen_ai.user.message` | Problem statement text | At start of agent run | -| `gen_ai.agent.finish` | JSON: exit status + total tokens | At end of agent run | - -**`chat` log records:** - -| `event_name` | `body` content | When | -|-------------|---------------|------| -| `gen_ai.system.message` | Full system prompt | For each system message in input | -| `gen_ai.user.message` | Full user message | For each user message in input | -| `gen_ai.assistant.message` | Prior assistant response | For each assistant message in input (multi-turn) | -| `gen_ai.tool.message` | Tool result text | For each tool result message in input | -| `gen_ai.choice` | Full LLM response text | For each response choice | -| `gen_ai.tool.call` | Tool call JSON (name + arguments) | For each tool call in the response | -| `gen_ai.thinking` | Full reasoning/thinking text | For each thinking block (Claude, DeepSeek, etc.) | - -**`execute_tool` log records:** - -| `event_name` | `body` content | When | -|-------------|---------------|------| -| `gen_ai.tool.input` | Tool command/action string | Before tool execution | -| `gen_ai.tool.output` | Tool observation/result | After tool execution | - -## OTel Collector Configuration - -The OTel Collector sits between your application and the backend. It receives OTLP data and routes traces and logs to separate backend streams. - -### Minimal Configuration +## What Parseable stores + +A single agent run produces one trace with three span types: + +- `invoke-agent` for the full run +- `chat {model}` for each LLM call +- `execute_tool {tool.name}` for each tool execution + +The messages and interaction details are kept as events on those spans. Parseable stores: + +- **span rows** for aggregate fields such as model, provider, token counts, duration, and status +- **event rows** for the detailed interaction content such as prompts, responses, tool input, tool output, reasoning, and evaluation results + +That is why the invocation view can show both a high-level run summary and a detailed drill-down without a separate GenAI logs pipeline. + +## OTel Collector configuration + +The collector sits between your application and Parseable. For this flow, one traces pipeline is enough. + +### Minimal configuration ```yaml receivers: @@ -689,7 +571,6 @@ processors: send_batch_size: 100 exporters: - # Traces -> Parseable otlphttp/traces: endpoint: https://your-parseable-instance:8000 encoding: json @@ -698,501 +579,104 @@ exporters: X-P-Stream: "genai-traces" X-P-Log-Source: "otel-traces" - # Logs -> Parseable - otlphttp/logs: - endpoint: https://your-parseable-instance:8000 - encoding: json - headers: - Authorization: "Basic " - X-P-Stream: "genai-logs" - X-P-Log-Source: "otel-logs" - service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlphttp/traces] - logs: - receivers: [otlp] - processors: [batch] - exporters: [otlphttp/logs] -``` - -### Key Points - -- **Two pipelines:** traces and logs are separate OpenTelemetry signals. The collector routes them independently. -- **One receiver:** both signals arrive at the same OTLP endpoint from the SDK. -- **JSON encoding:** the collector converts OpenTelemetry protobuf to JSON before sending to the backend. This is what the backend flattens into queryable records. -- **Batch processor:** accumulates records and sends them in batches. Tune `timeout` and `send_batch_size` for your throughput. For development, lower values (1s, 10 records) give faster feedback. For production, higher values reduce network overhead. - -### Running the Collector - -Docker: - -```bash -docker run -d --name otel-collector \ - -p 4317:4317 -p 4318:4318 \ - -v $(pwd)/otel-collector-config.yaml:/etc/otelcol/config.yaml \ - otel/opentelemetry-collector-contrib:latest ``` -Binary: +### Key points -```bash -otelcol-contrib --config otel-collector-config.yaml -``` - -## Semantic Conventions - -All attribute names follow the [OpenTelemetry GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). This matters because backends, dashboards, and tools that understand these conventions will automatically recognize and display GenAI data correctly. - -### Span Attributes by Span Type - -**`invoke_agent` spans:** - -| Attribute | Type | Required | Description | -|-----------|------|----------|-------------| -| `gen_ai.operation.name` | string | Yes | Always `"invoke_agent"` | -| `gen_ai.agent.name` | string | Yes | Agent identifier | -| `gen_ai.provider.name` | string | Yes | LLM provider: `"openai"`, `"anthropic"`, `"google"`, etc. | -| `gen_ai.request.model` | string | Yes | Model name used by the agent | -| `gen_ai.usage.input_tokens` | int | On completion | Total prompt tokens across all LLM calls | -| `gen_ai.usage.output_tokens` | int | On completion | Total completion tokens across all LLM calls | -| `gen_ai.response.finish_reasons` | string | On completion | JSON array: `["exit_command"]`, `["stop"]`, etc. | - -**`chat` spans:** - -| Attribute | Type | Required | Description | -|-----------|------|----------|-------------| -| `gen_ai.operation.name` | string | Yes | Always `"chat"` for chat completions | -| `gen_ai.provider.name` | string | Yes | LLM provider: `"openai"`, `"anthropic"`, `"google"`, etc. | -| `gen_ai.request.model` | string | Yes | Model name as passed to the API | -| `gen_ai.response.model` | string | On success | Model name as returned by the API (may differ from request) | -| `gen_ai.response.id` | string | On success | Provider-assigned response ID | -| `gen_ai.usage.input_tokens` | int | On success | Prompt token count for this call | -| `gen_ai.usage.output_tokens` | int | On success | Completion token count for this call | -| `gen_ai.request.temperature` | float | If set | Sampling temperature | -| `gen_ai.request.top_p` | float | If set | Nucleus sampling parameter | -| `gen_ai.request.max_tokens` | int | If set | Maximum output tokens | -| `gen_ai.response.finish_reasons` | string | On success | JSON array of finish reasons: `["stop"]`, `["tool_calls"]` | -| `gen_ai.tool.call.id` | string | If tool calls present | Tool call ID(s) for cross-span correlation with `execute_tool` spans. Single ID as string; multiple IDs as JSON array. | -| `error.type` | string | On error | Exception class name | - -**`execute_tool` spans:** - -| Attribute | Type | Required | Description | -|-----------|------|----------|-------------| -| `gen_ai.operation.name` | string | Yes | Always `"execute_tool"` | -| `gen_ai.tool.name` | string | Yes | Tool/command name (e.g., `"find_file"`, `"open_file"`) | -| `gen_ai.tool.type` | string | Yes | Always `"function"` | -| `gen_ai.tool.call.id` | string | If available | Tool call ID from function calling mode (links to LLM's tool call request) | -| `error.type` | string | On error | Exception class name (e.g., `"CommandTimeoutError"`) | - - -The older `gen_ai.system` attribute has been replaced by `gen_ai.provider.name` per the current OTel GenAI semantic conventions. - - -### Log Record Event Names - -**`chat` span events:** - -| `event_name` | Semantic Convention | Description | -|-------------|-------------------|-------------| -| `gen_ai.system.message` | GenAI message event | System prompt | -| `gen_ai.user.message` | GenAI message event | User input | -| `gen_ai.assistant.message` | GenAI message event | Prior assistant response (multi-turn) | -| `gen_ai.tool.message` | GenAI message event | Tool/function result | -| `gen_ai.choice` | GenAI choice event | LLM response content | -| `gen_ai.tool.call` | GenAI tool call event | Tool/function invocation | -| `gen_ai.thinking` | Custom extension | Reasoning/thinking block (not yet in OTel semconv) | - -**`invoke_agent` span events:** - -| `event_name` | Semantic Convention | Description | -|-------------|-------------------|-------------| -| `gen_ai.user.message` | GenAI message event | Problem statement that triggered the agent run | -| `gen_ai.agent.finish` | Custom extension | Agent completion summary (exit status + total tokens) | - -**`execute_tool` span events:** - -| `event_name` | Semantic Convention | Description | -|-------------|-------------------|-------------| -| `gen_ai.tool.input` | Custom extension | Tool command/action sent for execution | -| `gen_ai.tool.output` | Custom extension | Tool observation/result returned from execution | - -### Log Record Attributes - -Every log record carries: - -| Attribute | Type | Description | -|-----------|------|-------------| -| `gen_ai.operation.name` | string | `"chat"`, `"invoke_agent"`, or `"execute_tool"`. Identifies which span type emitted this log. | -| `gen_ai.event.name` | string | Duplicates `event_name` for queryability as a flat column | - -**Additional attributes on `chat` log records:** - -| Attribute | Type | Description | -|-----------|------|-------------| -| `gen_ai.provider.name` | string | LLM provider (`openai`, `anthropic`, `google`, `mistral`, etc.) | -| `gen_ai.request.model` | string | Model name as requested | -| `gen_ai.request.temperature` | float | Temperature (if set) | -| `gen_ai.request.top_p` | float | Top-p (if set) | -| `gen_ai.request.max_tokens` | int | Max output tokens (if set) | - -**Additional attributes on `invoke_agent` log records:** - -| Attribute | Type | Description | -|-----------|------|-------------| -| `gen_ai.provider.name` | string | LLM provider | -| `gen_ai.request.model` | string | Model name | -| `gen_ai.agent.name` | string | Agent identifier (e.g., `"swe-agent"`) | -| `role` | string | `"user"` (on `gen_ai.user.message` only) | -| `gen_ai.usage.input_tokens` | int | Total input tokens (on `gen_ai.agent.finish` only) | -| `gen_ai.usage.output_tokens` | int | Total output tokens (on `gen_ai.agent.finish` only) | - -**Additional attributes on `execute_tool` log records:** - -| Attribute | Type | Description | -|-----------|------|-------------| -| `gen_ai.tool.name` | string | Tool/command name | -| `gen_ai.tool.type` | string | Always `"function"` | -| `gen_ai.tool.call.id` | string | Tool call ID (empty string if not using function calling) | - -**Event-specific attributes (chat logs only):** - -| Attribute | Present on | Type | Description | -|-----------|-----------|------|-------------| -| `role` | `gen_ai.{role}.message` | string | `system`, `user`, `assistant`, `tool` | -| `index` | `gen_ai.choice` | int | Choice index (0-based) | -| `finish_reason` | `gen_ai.choice` | string | `stop`, `tool_calls`, `length`, etc. | -| `gen_ai.tool.name` | `gen_ai.tool.call` | string | Function/tool name | -| `gen_ai.tool.call.id` | `gen_ai.tool.call` | string | Tool call ID | - -### The body Field - -The `body` of each log record is the **full, untruncated content**: - -**`chat` span logs:** - -| Event | `body` contains | -|-------|----------------| -| `gen_ai.system.message` | Complete system prompt (can be thousands of tokens) | -| `gen_ai.user.message` | Complete user message | -| `gen_ai.assistant.message` | Complete prior assistant response | -| `gen_ai.tool.message` | Complete tool result | -| `gen_ai.choice` | Complete LLM response text | -| `gen_ai.tool.call` | JSON: `{"name": "...", "arguments": "..."}` | -| `gen_ai.thinking` | Complete reasoning/thinking text | - -**`invoke_agent` span logs:** +- one OTLP receiver is enough +- one traces pipeline is enough for agent observability in this model +- events are exported together with their parent spans +- Parseable receives the trace payload and flattens both span and event data into queryable records -| Event | `body` contains | -|-------|----------------| -| `gen_ai.user.message` | Complete problem statement that triggered the agent run | -| `gen_ai.agent.finish` | JSON: `{"exit_status": "...", "total_input_tokens": N, "total_output_tokens": N}` | - -**`execute_tool` span logs:** +## Semantic conventions -| Event | `body` contains | -|-------|----------------| -| `gen_ai.tool.input` | Complete tool command/action string | -| `gen_ai.tool.output` | Complete tool observation/result | +This guide follows the [OpenTelemetry GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai), which now live in the dedicated GenAI semantic conventions repository. -No truncation. No size limits from the instrumentation side. The body carries whatever the LLM returned, whatever was sent to it, or whatever the tool produced. +Use standard `gen_ai.*` span attributes wherever they apply. When you need to preserve prompt, response, tool, reasoning, or evaluation content, attach it as an event inside the active span so it stays tied to the exact part of the run that produced it. ## Scenarios -### Simple Chat Completion (OpenAI) - -```python -response = openai.chat.completions.create( - model="gpt-4o", - messages=[ - {"role": "system", "content": "You are a helpful assistant."}, - {"role": "user", "content": "What is 2+2?"}, - ], -) -``` - -**Produces:** -- 1 span: `chat gpt-4o` with token counts, latency, status OK -- 3 log records: `gen_ai.system.message`, `gen_ai.user.message`, `gen_ai.choice` -- All share the same `trace_id` + `span_id` - -### Multi-Turn Conversation - -```python -messages = [ - {"role": "system", "content": "You are a math tutor."}, - {"role": "user", "content": "What is 10*5?"}, - {"role": "assistant", "content": "50"}, - {"role": "user", "content": "Divide by 2"}, -] -response = call_llm("gpt-4o", messages) -``` - -**Produces:** -- 1 span: `chat gpt-4o` -- 5 log records: `gen_ai.system.message`, `gen_ai.user.message`, `gen_ai.assistant.message`, `gen_ai.user.message`, `gen_ai.choice` -- The full conversation history is captured. You can reconstruct it by querying log records for this span, ordered by timestamp. - -### Tool/Function Calling - -The LLM responds with tool calls instead of (or in addition to) text content. - -**Produces:** -- 1 span: `chat gpt-4o`, `finish_reasons: ["tool_calls"]` -- N input message logs -- 1 choice log (may have empty body if the LLM only returned tool calls) -- 1+ `gen_ai.tool.call` logs, each with the tool name and JSON arguments in `body` - -### Claude Thinking/Reasoning Blocks - -Claude (and some other models) return a `thinking_blocks` array alongside the response content. Each thinking block contains the model's chain-of-thought reasoning. - -**Produces:** -- 1 span: `chat claude-sonnet-4-20250514` -- N input message logs -- 1 choice log (the actual response text) -- 1+ `gen_ai.thinking` logs, each with the full reasoning text in `body` - -This is one of the main reasons to instrument the agent yourself. Thinking blocks are provider-specific, so capture them explicitly as `gen_ai.thinking` logs when your model exposes them. - -### Error: Rate Limit - -```python -try: - response = call_llm(...) -except RateLimitError as e: - # The span still exists with ERROR status - pass -``` - -**Produces:** -- 1 span: `chat gpt-4o`, `span_status=ERROR`, `error.type=RateLimitError` -- N input message logs (emitted before the call, so they still exist) -- 0 choice/tool/thinking logs (call failed before response) -- All logs still have `trace_id` + `span_id`, so you can see what was sent to the LLM even though it failed - -### Error: Context Window Exceeded - -The LLM returns a 400 because the input is too long. - -**Produces:** -- Same as above: span with ERROR status, input message logs but no response logs. -- `error.type=ContextWindowExceededError` -- This is queryable: find all spans where `error.type = 'ContextWindowExceededError'`, then JOIN with logs to see what input caused it. - -### Streaming Responses - -**Current limitation:** The instrumentation code shown above works with non-streaming responses. For streaming (`stream=True`), you need to: +### Simple chat completion -1. Open the span before the stream starts -2. Accumulate the streamed chunks into a full response -3. Emit log records and set span attributes after the stream completes -4. Close the span +What you should expect: -The span remains open during streaming, so all log records emitted during or after streaming will be correlated. +- one `chat` span +- one event for each input message +- one `gen_ai.choice` event for the response +- optionally one `gen_ai.client.inference.operation.details` event for the full request and response -```python -with _tracer.start_as_current_span(f"chat {model}", kind=trace.SpanKind.CLIENT) as span: - span.set_attribute("gen_ai.request.model", model) - # ... emit input message logs ... - - stream = client.chat.completions.create(model=model, messages=messages, stream=True) - chunks = [] - for chunk in stream: - chunks.append(chunk) - - # Reconstruct full response from chunks, then emit logs - full_response_text = "".join(c.choices[0].delta.content or "" for c in chunks if c.choices) - _otel_logger.emit(body=full_response_text, event_name="gen_ai.choice", ...) - - # Token usage may not be available in streaming mode (provider-dependent) - span.set_status(trace.StatusCode.OK) -``` - -### Retries - -If your agent retries failed LLM calls (e.g., on rate limit errors), each attempt produces its own span. The retry loop should be **outside** the span context: - -```python -for attempt in range(max_retries): - try: - response = call_llm(model, messages) # Each call creates its own span - break - except RateLimitError: - time.sleep(backoff) -``` - -This produces N spans (one per attempt), each with its own status. Failed attempts have `ERROR` status, the successful attempt has `OK`. - -### Multiple LLM Providers - -The `gen_ai.provider.name` attribute distinguishes providers. If your agent calls OpenAI for some tasks and Anthropic for others: - -- OpenAI calls: `gen_ai.provider.name = "openai"` -- Anthropic calls: `gen_ai.provider.name = "anthropic"` - -You can filter and aggregate by provider in queries. - -### Wrapper Libraries (LiteLLM, LangChain) - -If your agent uses a wrapper library like LiteLLM that abstracts over multiple providers: - -- Instrument at the **wrapper call site**, not inside the wrapper library -- The model name you pass to the wrapper becomes `gen_ai.request.model` -- The `gen_ai.provider.name` should reflect the actual underlying provider (if known) -- The response object structure may differ from the raw OpenAI SDK, so adapt the attribute extraction accordingly - -## Limitations and Edge Cases - -### Thinking Blocks Are Provider-Specific - -The `gen_ai.thinking` event is not part of the official OTel GenAI semantic conventions. It is a custom extension. Different providers expose reasoning differently: - -| Provider | How thinking is exposed | How to extract | -|----------|----------------------|----------------| -| Anthropic (Claude) | `response.choices[i].message.thinking_blocks` (via LiteLLM) | Iterate `thinking_blocks`, extract `"thinking"` key | -| DeepSeek | `response.choices[i].message.reasoning_content` | Extract `reasoning_content` field | -| OpenAI o1/o3 | Internal reasoning not exposed in API response | Cannot be captured | - -If the provider does not expose reasoning, there will be no `gen_ai.thinking` log records. This is expected behavior, not a bug. - -### Token Counts May Be Absent +### Multi-turn conversation -Some scenarios where `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` are not available: +What you should expect: -| Scenario | Why | Impact | -|----------|-----|--------| -| Error spans | API call failed before returning usage | Span has no token attributes | -| Streaming without usage chunks | Some providers don't send usage in streaming mode | Span has no token attributes | -| Local/self-hosted models | Some local model servers don't return usage | Span has no token attributes | -| Wrapper library filtering | Some wrappers strip usage from the response object | Span has no token attributes | +- one `chat` span for the call +- one span event for each system, user, assistant, or tool message that was part of the input +- one or more `gen_ai.choice` events for the returned answer +- optionally one `gen_ai.client.inference.operation.details` event when your SDK emits the full structured operation details -**Backend impact:** Token-based aggregations should handle NULL gracefully. +### Tool calling -### The body Field Can Be Very Large +What you should expect: -System prompts in agent applications can be 10,000+ tokens. LLM responses can be similarly large. Tool call arguments can contain large JSON payloads. Thinking blocks can be tens of thousands of tokens. +- one `chat` span +- one or more `gen_ai.tool.call` events when the model asks for tools +- one `execute_tool` span per actual tool execution +- `gen_ai.tool.input` and `gen_ai.tool.output` events on the tool span -There is **no truncation** in the instrumentation. The full content is emitted as the log record body. This is intentional because truncation would make conversation reconstruction incomplete. +### Thinking or reasoning blocks -**Backend impact:** The logs stream will be significantly larger (in bytes) than the traces stream. Plan storage and indexing accordingly. +If the provider exposes reasoning blocks, attach them as `gen_ai.thinking` span events on the active chat span. -### Log Records Without Span Context +### Evaluation results -If `_otel_logger.emit()` is called **outside** a `start_as_current_span()` context, the log record will have empty `trace_id` and `span_id`. This means: +If you evaluate the final response, attach the result as `gen_ai.evaluation.result`. Parent it to the span being evaluated when possible. For example, attach a relevance or sentiment score to the final `chat {model}` span, or attach an end-to-end success score to the `invoke-agent` span. -- The log record exists in the logs stream but cannot be correlated with any span -- This is a bug in the instrumentation code. All GenAI log emissions should be inside a span context. - -The manual instrumentation approach prevents this by design: all `emit()` calls are inside the `with _tracer.start_as_current_span(...):` block. - -### Concurrent/Async LLM Calls - -OTel context propagation is thread-local (and async-task-local in asyncio). If your agent makes concurrent LLM calls: - -- **Threading:** Each thread has its own context. Spans in different threads don't interfere. Each call gets its own `trace_id`/`span_id`. -- **asyncio:** OTel SDK supports async context propagation via `contextvars`. Each coroutine gets its own context. -- **Multiprocessing:** Each process has its own TracerProvider/LoggerProvider. Spans from different processes have different `trace_id`s. - -Concurrency does not break correlation. Each LLM call's logs are always linked to that call's span. - -### Provider-Specific Response Formats - -Different LLM providers return responses in slightly different formats. The instrumentation code must handle these differences: - -| Provider | `response.model` | `response.usage` | Tool calls | Thinking | -|----------|-----------------|------------------|------------|----------| -| OpenAI | Always present | Always present (non-streaming) | `.message.tool_calls` | N/A | -| Anthropic (via LiteLLM) | Always present | Always present | `.message.tool_calls` | `.message.thinking_blocks` | -| Google (via LiteLLM) | Always present | Always present | `.message.tool_calls` | N/A | -| Local models (Ollama, vLLM) | May differ | May be absent | Varies | N/A | - -Guard all attribute extraction with `hasattr()` / `getattr()` / `is not None` checks. - -### The opentelemetry-instrument CLI Must Wrap the Process - -The CLI sets up providers at process startup. If your application forks or spawns subprocesses, those subprocesses will NOT have the providers configured. Each process that emits telemetry must be launched with `opentelemetry-instrument`, or must set up providers manually. - -### Collector Must Be Running - -If the OTel Collector is not running when the application starts: - -- The SDK will buffer spans and log records in memory -- It will periodically retry exporting -- If the buffer fills up, the oldest records are dropped (configurable via `OTEL_BSP_MAX_QUEUE_SIZE`, default 2048 for spans) -- No application errors are raised, so telemetry loss is silent - -For production, ensure the collector is running before the application starts. For development, data loss during collector restarts is acceptable. - -### Scope Name Consistency - -Both `_tracer` and `_otel_logger` should use the **same scope name**. This makes it easy to filter both spans and logs by scope: +### Errors -```python -_tracer = trace.get_tracer("my-agent.llm", "1.0.0") -_otel_logger = get_logger_provider().get_logger("my-agent.llm", "1.0.0") -``` +If the call fails before a response is returned, you should still see: -If they use different scope names, correlation still works (via `trace_id`/`span_id`), but filtering by `scope_name` in queries will not match both signals. +- the span still exists +- the span carries `error.type` +- any prompt or input events emitted before the failure still remain visible in the trace ## Troubleshooting -### No spans or logs appearing - -1. **Is the collector running?** Check `curl http://localhost:4318/v1/traces`. It should return a response, even an error response means it is listening. -2. **Is the CLI wrapping the process?** Run `opentelemetry-instrument --help` to verify it's installed. Check that your launch command uses `opentelemetry-instrument python ...`, not just `python ...`. -3. **Are exporters configured?** Check that `--traces_exporter otlp --logs_exporter otlp` are passed to the CLI. -4. **Is the endpoint correct?** `OTEL_EXPORTER_OTLP_ENDPOINT` must match where the collector is listening. - -### Duplicate spans per LLM call - -Check whether your application or framework is creating additional spans around the same LLM request. Keep one `chat {model}` span for each LLM call, and remove any extra SDK middleware or framework tracing that emits duplicate HTTP, template, database, or model-client spans for the same operation. - -### Log records have empty trace_id/span_id +### No traces appearing -Log records are being emitted outside a span context. Ensure every `_otel_logger.emit()` call is inside a `with _tracer.start_as_current_span(...):` block. +1. check that the collector is running +2. check that the process is launched with `opentelemetry-instrument` +3. check that `--traces_exporter otlp` is set +4. check that `OTEL_EXPORTER_OTLP_ENDPOINT` points to the collector -### "Overriding of current LoggerProvider is not allowed" +### Span exists but event detail is missing -You are calling `set_logger_provider()` in your application code, but the CLI already set up a LoggerProvider. Remove the manual `set_logger_provider()` call. Use `get_logger_provider()` instead. +The span likely exported correctly, but the message, tool, or reasoning content was never added as a span event. Check that `span.add_event(...)` is called inside the active span. -### Spans appear but logs do not +### Too many duplicate chat spans -1. Check that `--logs_exporter otlp` is passed to the CLI (not `none`). -2. Check that the collector config has a `logs` pipeline (not just `traces`). -3. Verify that `_otel_logger.emit()` is being called. Add a `print()` before the emit to confirm the code path is reached. +Keep one `chat {model}` span per model call. Remove extra middleware or framework tracing around the same operation if it creates duplicates. -### Logs appear but with wrong event_name +### Provider setup errors -The `event_name` parameter in `_otel_logger.emit()` must be a keyword argument. If passed positionally, it may be interpreted as a different parameter. Always use `event_name=...`. +If you are using the CLI wrapper, do not manually create your own exporter pipeline unless you specifically need a custom setup. Let the CLI configure the `TracerProvider`, then only use `trace.get_tracer(...)` in your code. -## Verification Checklist +## Verification checklist -After setting up instrumentation, verify end-to-end: +After setup, verify the flow end to end. You should be able to open one invocation and understand the run without having to mentally stitch separate signals together. | Check | How | Expected | |-------|-----|----------| -| **Three span types present** | Query traces and check `gen_ai.operation.name` values | `invoke_agent`, `chat`, `execute_tool` all present | -| **Span hierarchy correct** | Pick a trace. Check that `chat` and `execute_tool` spans have `parent_span_id` matching the `invoke_agent` span's `span_id` | All child spans point to the `invoke_agent` parent | -| **One `invoke_agent` per run** | Count `invoke_agent` spans | 1 per agent run | -| **One `chat` span per LLM call** | Count `chat` spans vs. LLM calls made | 1:1 ratio | -| **One `execute_tool` per tool exec** | Count `execute_tool` spans vs. tool executions | 1:1 ratio | -| **`gen_ai.agent.name` on invoke_agent** | Check span attributes | Present on `invoke_agent` span, NOT on `chat` spans | -| **`gen_ai.provider.name` on chat spans** | Check span for `gen_ai.provider.name` (not `gen_ai.system`) | Present on every `chat` span | -| **`gen_ai.tool.name` on execute_tool** | Check span attributes | Tool name present | -| **Request attributes present** | Check `chat` span for `gen_ai.request.model`, `gen_ai.provider.name` | Present on every `chat` span | -| **Response attributes present** | Check `chat` span for `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` | Present on successful `chat` spans | -| **Aggregate tokens on invoke_agent** | Check `invoke_agent` span for `gen_ai.usage.input_tokens` | Total across all LLM calls | -| **Error spans captured** | Trigger a timeout. Check for `execute_tool` span with `span_status_code = 2`. | Span exists with `error.type` | -| **Log records exist** | Query logs for same `trace_id` as an `invoke_agent` span | Multiple log records | -| **Trace-log correlation** | Pick a `chat` span. Query logs where `trace_id` and `span_id` match. | Logs link to the correct `chat` span | -| **All chat event types present** | Check `event_name` values on `chat` logs | `gen_ai.system.message`, `gen_ai.user.message`, `gen_ai.choice`, and optionally `gen_ai.tool.call`, `gen_ai.thinking` | -| **invoke_agent logs present** | Check logs with `gen_ai.operation.name = "invoke_agent"` | `gen_ai.user.message` (problem statement) and `gen_ai.agent.finish` (completion summary) | -| **execute_tool logs present** | Check logs with `gen_ai.operation.name = "execute_tool"` | `gen_ai.tool.input` and `gen_ai.tool.output` paired for each tool execution | -| **All three operation types in logs** | Group logs by `gen_ai.operation.name` | `invoke_agent`, `chat`, and `execute_tool` all present | -| **Body is untruncated** | Check `body` field on a `gen_ai.choice` log record | Full LLM response text, not truncated | -| **No duplicates** | Check that each LLM call produces exactly 1 `chat` span and 1 `gen_ai.choice` log per response choice | No doubles | +| Three span types present | Query traces and inspect `gen_ai.operation.name` | `invoke_agent`, `chat`, `execute_tool` | +| Span hierarchy correct | Open a trace and compare parent-child relationships | `chat` and `execute_tool` spans sit under `invoke-agent` | +| Request fields present | Check `chat` span attributes | model, provider, request config present | +| Token fields present | Check successful `chat` spans | input and output token fields present | +| Tool fields present | Check `execute_tool` span attributes | tool name, tool type, optional tool call ID | +| Event rows present | Open the same trace and inspect event rows | prompt, response, tool I/O, reasoning, or evaluation events visible | +| Errors captured | Trigger a failed tool or model call | span has error status and `error.type` | +| No duplicate spans | Compare one user-visible step to the trace | one logical operation maps to one span | diff --git a/content/docs/user-guide/agent-observability/schema-reference.mdx b/content/docs/user-guide/agent-observability/schema-reference.mdx index 9cd90fd..da3c1e0 100644 --- a/content/docs/user-guide/agent-observability/schema-reference.mdx +++ b/content/docs/user-guide/agent-observability/schema-reference.mdx @@ -2,7 +2,7 @@ title: Schema Reference --- -When Parseable receives GenAI traces at the `/v1/traces` endpoint, it flattens the nested OTel JSON into a single row per span event. This page documents every column you can query. +When Parseable receives GenAI traces at the `/v1/traces` endpoint, it flattens the nested OTel JSON into queryable rows for both spans and span events. This page documents the columns you are most likely to query. ## Span Hierarchy @@ -11,22 +11,24 @@ A single agent invocation produces a tree of spans sharing the same `span_trace_ ``` span_trace_id: abc123 | -+-- [root] agent_run (gen_ai.operation.name = "agent") ++-- [root] invoke-agent (gen_ai.operation.name = "invoke_agent") | +-- chat gpt-4o (gen_ai.operation.name = "chat") - | +-- event: gen_ai.prompt (event_name row) - | +-- event: gen_ai.completion (event_name row) + | +-- event: gen_ai.user.message + | +-- event: gen_ai.choice | +-- execute_tool search_api (gen_ai.operation.name = "execute_tool") + | +-- event: gen_ai.tool.input + | +-- event: gen_ai.tool.output | +-- chat gpt-4o (gen_ai.operation.name = "chat") - +-- event: gen_ai.prompt (event_name row) - +-- event: gen_ai.completion (event_name row) + +-- event: gen_ai.user.message + +-- event: gen_ai.choice ``` Key points: - **Span rows** have `event_name IS NULL`. These contain the `gen_ai.*` attributes, token counts, and Parseable-enriched columns. -- **Event rows** have `event_name` set (e.g., `gen_ai.prompt` or `gen_ai.completion`). These contain the message content in `event_gen_ai.prompt` or `event_gen_ai.completion`. Event rows are only present when content capture is enabled (`OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true`). +- **Event rows** have `event_name` set (e.g., `gen_ai.client.inference.operation.details`, `gen_ai.user.message`, `gen_ai.choice`, `gen_ai.tool.output`, or `gen_ai.evaluation.result`). Parseable flattens event attributes with an `event_` prefix, so event content and event metadata stay queryable alongside the parent span. - Use `span_parent_span_id` to reconstruct the call tree within a trace. --- @@ -70,7 +72,7 @@ Columns that identify the GenAI operation being performed. | Column | Type | Description | |--------|------|-------------| -| `gen_ai.operation.name` | String | Type of operation: `chat`, `text_completion`, `embeddings`, `execute_tool`, `agent`. | +| `gen_ai.operation.name` | String | Type of operation: `chat`, `text_completion`, `embeddings`, `execute_tool`, `invoke_agent`. | | `gen_ai.system` | String | GenAI provider system identifier (e.g., `openai`, `anthropic`, `cohere`). | | `gen_ai.provider.name` | String | Provider name when different from system. | | `gen_ai.conversation.id` | String | Unique identifier for a multi-turn conversation thread. | @@ -122,7 +124,7 @@ Columns tracking token consumption per span. ## Agent -Columns specific to agent-level spans (where `gen_ai.operation.name = 'agent'`). +Columns specific to agent-level spans (where `gen_ai.operation.name = 'invoke_agent'`). | Column | Type | Description | |--------|------|-------------| @@ -146,15 +148,25 @@ Columns specific to tool call spans (where `gen_ai.operation.name = 'execute_too --- -## Content Columns (Opt-In) +## Event Columns -These columns are only populated when content capture is enabled via `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true`. They appear on **event rows** (where `event_name` is not null). +These columns appear on **event rows** where `event_name` is not null. The exact set depends on which event attributes your instrumentation emits. | Column | Type | Description | |--------|------|-------------| -| `event_name` | String | Event type: `gen_ai.prompt`, `gen_ai.completion`, `gen_ai.tool.message`, etc. | -| `event_gen_ai.prompt` | String | The full prompt or user message content. Present on `gen_ai.prompt` events. | -| `event_gen_ai.completion` | String | The full completion or assistant message content. Present on `gen_ai.completion` events. | +| `event_name` | String | Event type such as `gen_ai.client.inference.operation.details`, `gen_ai.user.message`, `gen_ai.choice`, `gen_ai.tool.call`, `gen_ai.tool.input`, `gen_ai.tool.output`, or `gen_ai.evaluation.result`. | +| `event_gen_ai.event.name` | String | Event name repeated as a GenAI attribute when your instrumentation emits `gen_ai.event.name`. This usually matches `event_name`. | +| `event_body` | String | Event payload when your instrumentation emits a `body` attribute. This is useful for prompt text, tool input, tool output, finish summaries, and reasoning text. | +| `event_content` | String | Full content emitted with the event when your instrumentation uses a `content` field. | +| `event_role` | String | Message role when present, for example `user`, `assistant`, `system`, or `tool`. | +| `event_gen_ai.input.messages` | String | Input messages when emitted through the OpenTelemetry GenAI operation details event. | +| `event_gen_ai.output.messages` | String | Output messages when emitted through the OpenTelemetry GenAI operation details event. | +| `event_gen_ai.tool.name` | String | Tool name on tool-related events when emitted. | +| `event_gen_ai.tool.call.id` | String | Tool call identifier on related events when emitted. | +| `event_gen_ai.evaluation.name` | String | Name of the evaluator or evaluation metric when `gen_ai.evaluation.result` is emitted. | +| `event_gen_ai.evaluation.score.label` | String | Human-readable evaluation label, for example `satisfied`, `neutral`, `correct`, or `fail`. | +| `event_gen_ai.evaluation.score.value` | Float | Numeric evaluation score when available. | +| `event_gen_ai.evaluation.explanation` | String | Free-form explanation for the evaluation result when available. | | `event_time_unix_nano` | BigInt | Event timestamp in nanoseconds since epoch. | | `event_dropped_attributes_count` | Int | Number of event attributes dropped due to limits. |