feat(ai): add agent hooks and OpenTelemetry tracing - #1536
Open
chaojixinren wants to merge 1 commit into
Open
Conversation
Contributor
|
@ambiguous-pointer please help review this PR. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds configurable agent lifecycle hooks and OpenTelemetry tracing across the ReAct, SSE, MCP, and runtime layers.
Changes:
- Introduces hook management, logging, OTLP tracing, and content-capture policies.
- Instruments agent/model/tool lifecycles with propagation and fallback metadata.
- Adds shutdown handling, configuration, and extensive tests.
Reviewed changes
Copilot reviewed 39 out of 40 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
ai/test/e2e/e2e_test.go |
Wires hooks into E2E runtime. |
ai/schema/json/hooks.schema.json |
Defines hooks configuration schema. |
ai/main.go |
Registers hooks and adjusts lifecycle order. |
ai/go.sum |
Records HTTP OTLP dependencies. |
ai/go.mod |
Adds OpenTelemetry dependencies. |
ai/config/test/loader_test.go |
Tests hooks configuration defaults. |
ai/config/loader.go |
Maps hooks to its schema. |
ai/config.yaml |
Enables the hooks component. |
ai/component/tools/engine/mcp_tools.go |
Propagates trace headers to MCP. |
ai/component/tools/engine/mcp_tools_test.go |
Tests outbound trace propagation. |
ai/component/server/engine/sse/sse.go |
Centralizes CORS handling. |
ai/component/server/engine/router.go |
Allows and exposes tracing headers. |
ai/component/server/engine/router_test.go |
Tests tracing CORS headers. |
ai/component/server/engine/handlers.go |
Propagates context and trace IDs. |
ai/component/server/engine/handlers_test.go |
Tests context and detached output handling. |
ai/component/hooks/tracing.go |
Implements lifecycle span creation. |
ai/component/hooks/tracing_test.go |
Tests spans, attributes, and capture. |
ai/component/hooks/README.md |
Documents hooks and OTLP setup. |
ai/component/hooks/manager.go |
Implements hook registration and dispatch. |
ai/component/hooks/manager_test.go |
Tests manager behavior and concurrency. |
ai/component/hooks/jaeger_e2e_test.go |
Adds optional Jaeger verification. |
ai/component/hooks/hooks.yaml |
Provides default hooks configuration. |
ai/component/hooks/factory.go |
Adds the hooks factory. |
ai/component/hooks/event.go |
Defines lifecycle events and snapshots. |
ai/component/hooks/component.go |
Implements hooks component lifecycle. |
ai/component/hooks/component_test.go |
Tests configuration and shutdown. |
ai/component/agent/react/steps.go |
Instruments stages, models, and tools. |
ai/component/agent/react/step_test.go |
Tests fallback and hook emissions. |
ai/component/agent/react/react.go |
Adds interaction tracing and draining. |
ai/component/agent/react/prompt.go |
Retains stage names and models. |
ai/component/agent/react/page_context_test.go |
Updates context construction test. |
ai/component/agent/react/orchestrator.go |
Instruments iterations and stages. |
ai/component/agent/react/orchestrator_test.go |
Tests lifecycle sequencing. |
ai/component/agent/react/lifecycle_test.go |
Tests cancellation and concurrency. |
ai/component/agent/react/hook_content.go |
Converts messages for telemetry. |
ai/component/agent/react/hook_content_test.go |
Tests semantic message conversion. |
ai/component/agent/react/component.go |
Connects hooks and agent lifecycle. |
ai/component/agent/react/component_wiring_test.go |
Tests hooks manager wiring. |
ai/component/agent/fallback/handler.go |
Reports fallback parsing usage. |
ai/component/agent/agent.go |
Adds context-aware interaction channels. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
chaojixinren
force-pushed
the
feat/hook-system
branch
from
August 22, 2026 06:46
7bbcab1 to
2acf02b
Compare
ambiguous-pointer
left a comment
ambiguous-pointer
left a comment
Contributor
There was a problem hiding this comment.
@chaojixinren 上述是我的一些个人拙见，可以按照您的设计进行实际的一些调整和修改 : )
Contributor
There was a problem hiding this comment.
Go 1.26 下 sonic v1.14.1 无法编译, 请先行合入远程更改
Comment on lines
+33
to
+44
| const ( | ||
| EventInteractionStart Event = "interaction.start" | ||
| EventInteractionEnd Event = "interaction.end" | ||
| EventIterationStart Event = "iteration.start" | ||
| EventIterationEnd Event = "iteration.end" | ||
| EventStageStart Event = "stage.start" | ||
| EventStageEnd Event = "stage.end" | ||
| EventModelCallStart Event = "model_call.start" | ||
| EventModelCallEnd Event = "model_call.end" | ||
| EventToolCallStart Event = "tool_call.start" | ||
| EventToolCallEnd Event = "tool_call.end" | ||
| ) |
Contributor
There was a problem hiding this comment.
-
潜在问题：
- 没有错误/降级/取消类事件。工具失败、observe 超时/解析失败、显式取消，在 PR 里都只是
State上的字段（Degraded/FallbackUsed/Error）。对 tracing/logging 两个消费方够用（span 上有error.type、agent.degraded属性），
但对未来的 metrics / 审计类 hook（它们需要"按事件种类订阅"）就缺了第一类公民：- 例：生产上想统计"每月 observe 超时次数"或"工具失败率"，现在只能订阅
stage.end然后过滤State.FallbackUsed，
而不是订阅一个语义明确的agent.degraded事件；若未来事件字段演进，这类消费方会静默错算。
- 例：生产上想统计"每月 observe 超时次数"或"工具失败率"，现在只能订阅
- 没有错误/降级/取消类事件。工具失败、observe 超时/解析失败、显式取消，在 PR 里都只是
-
个人建议: 事件种类扩展为
agent/stage/llm/tool × start/end/error+agent.degraded+
agent.cancel+llm.chunk（预留），并给State增加Seq uint64。事件字段只读约定保持。因为模型部署侧可能不一定都是稳定的模型，例如 VLLM 私有化部署的时候，工具调用参数模板没有绑定正确的时候，调用工具会出现偶发性的直接中断。所以会需要预设详细一些 -
生产场景：Dubbo 服务诊断场景（agent 通过 MCP 调
get_service_detail/诊断工具 feat: add PromQL and trace diagnosis tools #1499）——SRE 想要"按工具维度"的失败率报表，若没有独立tool.error事件种类，报表逻辑要散落在每个消费方里重复过滤，接入点越多越容易漏。
Comment on lines
+111
to
+161
| type lazyContentSnapshot struct { | ||
| once sync.Once | ||
| provider func() any | ||
| content string | ||
| } | ||
|
|
||
| func newLazyContentSnapshot(provider func() any) *lazyContentSnapshot { | ||
| if provider == nil { | ||
| return nil | ||
| } | ||
| return &lazyContentSnapshot{provider: provider} | ||
| } | ||
|
|
||
| func (s *lazyContentSnapshot) snapshot() string { | ||
| if s == nil { | ||
| return "" | ||
| } | ||
| s.once.Do(func() { | ||
| s.content = SnapshotContent(s.provider()) | ||
| s.provider = nil | ||
| }) | ||
| return s.content | ||
| } | ||
|
|
||
| // WithInputContent attaches an immutable input snapshot that is materialized | ||
| // only if a matching content-capturing hook requests it. | ||
| func (s State) WithInputContent(provider func() any) State { | ||
| s.inputContent = newLazyContentSnapshot(provider) | ||
| return s | ||
| } | ||
|
|
||
| // WithOutputContent attaches an immutable output snapshot that is materialized | ||
| // only if a matching content-capturing hook requests it. | ||
| func (s State) WithOutputContent(provider func() any) State { | ||
| s.outputContent = newLazyContentSnapshot(provider) | ||
| return s | ||
| } | ||
|
|
||
| func (s State) snapshotInputContent() string { | ||
| if s.Input != "" { | ||
| return s.Input | ||
| } | ||
| return s.inputContent.snapshot() | ||
| } | ||
|
|
||
| func (s State) snapshotOutputContent() string { | ||
| if s.Output != "" { | ||
| return s.Output | ||
| } | ||
| return s.outputContent.snapshot() | ||
| } |
Contributor
There was a problem hiding this comment.
- 潜在问题：
lazyContent字段（manager.go:44,96）只有包内NewTracingRegistration能用（未导出），
外部捕获内容的 hook 一律走manager.go:193-195的急切快照分支——也就是说"懒"只对内置 tracing hook 成立，对将来第三方内容型 hook（如审计）不成立。PR文档里"Content is serialized lazily"的表述容易误导。 - 个人建议：把
lazyContent语义并入公开的Registration（如CaptureContent: CaptureLazy）或至少在
Registration上注释清楚两档行为。 - 生产场景：审计 hook 需要"模型输入/输出原文"留档——如果它被急切序列化，每次模型调用都会多一次完整 JSON marshal（大对话可能几百 KB），生产热点路径上不可忽略；同时内容进内存=更大的 PII 暴露面。应能声明"延迟到真正落盘前才序列化"。
Comment on lines
74
to
78
| defer func() { | ||
| if r := recover(); r != nil { | ||
| sseHandler.HandleError("internal_error", fmt.Sprintf("internal error: %v", r)) | ||
| } | ||
| }() |
Contributor
There was a problem hiding this comment.
// ← 这里没有 go discardAgentOutput(channels)！
对比另外两条退出路径（handlers.go:104-107 和 134-136）
- handler 在流中途 panic（比如
MessageDelta遇到未知 final 类型、或任何将来加的代码）； - defer recover 触发 → 发一条
internal_errorSSE → handler 返回 → gin 关连接； - 交互 goroutine 不会死——
interactionCtx := context.WithoutCancel(extractedCtx)（handlers.go:87）已经把请求取消剥掉了，客户端断开对它是透明的； - 交互 goroutine 继续生成，调用
chans.Send（agent.go:62-70）：
func (chans *Channels) Send(sf *schema.StreamFeedback) {
sf.SetIndex(chans.nextIndex)
chans.nextIndex++
chans.UserRespChan <- sf // ← 有界阻塞发送，缓冲满就永久卡住
}- 缓冲（bufferSize）塞满 ~16 条后，没有任何人排空 → 交互 goroutine 永久阻塞在
Send； - 连锁反应：阻塞在
Send意味着 goroutine 的 defer 永远不执行——interaction.end事件发不出去（trace 缺尾）、finishInteraction不执行（ra.active表里的条目永不删除）； - 进程关闭时
Stop()（react.go:207-222）对这条交互执行activeWG.Wait()→ Stop 也跟着挂起，直到外层 20s 超时兜底，关闭质量劣化。
一句话：handler 侧一个 panic，产生一个永久阻塞的 goroutine + 一条不完整的 trace + Stop 挂起——而这一切本来用一行 go discardAgentOutput(channels) 就能避免。
根子是 Channels.Send 的阻塞语义——discardAgentOutput 只是防呆补丁，而且只在 handler 这一侧有
Contributor
There was a problem hiding this comment.
个人理解 这个和整个 agent 调用的 : 交互持久化 + 事件日志 + 整段重放 有关
可能得 #1534 完成后全面的思考一下这个地方如何实现
Contributor
There was a problem hiding this comment.
这里可能对于开发者或者需要基于钩子实现私有化能力的时候,会存在 我不知道有哪些 hook、不知道接入点
type: hooks
spec:
hooks:
- name: "logging"
enabled: true
events: ["agent.start", "agent.end", "agent.error", "agent.degraded", "agent.cancel",
"stage.start", "stage.end", "stage.error", "llm.start", "llm.end", "llm.error",
"tool.start", "tool.end", "tool.error"]
config: { level: "info" }
Contributor
There was a problem hiding this comment.
events 字段以 enum 形式列出全部事件种类
dubbo-admin/ai/schema/json/memory.schema.json
Lines 15 to 20 in 5d6df6e
chaojixinren
added a commit
to chaojixinren/dubbo-admin
that referenced
this pull request
Aug 23, 2026
Address review feedback from ambiguous-pointer: - Document that Registration.CaptureContent uses eager snapshots for external hooks; lazyContent (deferred serialization) is internal-only. README now says tracing defers "until IsRecording()", not "lazily" as a blanket statement (component/hooks/manager.go, README.md). - Add "Available events" section to README listing the current event vocabulary (interaction/iteration/stage/model_call/tool_call × start/end) and clarifying that State fields carry error/degraded/fallback metadata. - Add inline event reference to hooks.yaml so developers configuring custom hooks know what events exist without diving into code. Refs: apache#1536 (comment) apache#1536 (comment)
chaojixinren
added a commit
to chaojixinren/dubbo-admin
that referenced
this pull request
Aug 23, 2026
Address review feedback: sonic v1.14.1 fails to compile under Go 1.26.
Upgrade to v1.15.2 which resolves the compatibility issue.
Tested: go build ./... && go test -count=1 ./component/hooks/...
./component/server/engine/ pass on Go 1.25.7
Refs: apache#1536 (comment)
chaojixinren
added a commit
to chaojixinren/dubbo-admin
that referenced
this pull request
Aug 23, 2026
…hooks
Address review feedback from ambiguous-pointer: expand event vocabulary to
support first-class error, cancellation, and degradation events, enabling
future metrics and audit hooks to subscribe by semantic event type rather
than filtering State fields.
Changes:
- Add event types: interaction.error/.cancel/.degrade, stage.error,
model_call.error, tool_call.error
- Emit *.error events before the corresponding *.end when operations fail
- Emit interaction.cancel when context cancellation aborts an interaction
- Emit interaction.degrade when tool failures or fallback responses occur
- Add state.Degraded field to track tool call failures across the interaction
- Update documentation and inline event reference in hooks.yaml
Event flow examples:
- Successful model call: model_call.start → model_call.end
- Failed model call: model_call.start → model_call.error → model_call.end
- Tool degradation: tool_call.start → tool_call.error → tool_call.end
- Cancelled interaction: interaction.start → interaction.cancel → interaction.end
This preserves the existing State field approach (Error/Degraded/FallbackUsed)
while adding dedicated event types for consumers that need per-dimension
subscriptions (e.g., "all tool errors" or "interaction cancellations").
Tested: go test -count=1 ./component/hooks/... ./component/agent/react/...
all pass; existing tests cover the extended event emission paths
Refs: apache#1536 (comment)
chaojixinren
added a commit
to chaojixinren/dubbo-admin
that referenced
this pull request
Aug 23, 2026
Add lifecycle hooks infrastructure and OpenTelemetry tracing integration for the ReAct agent, enabling observability via structured logging and distributed tracing. ## Key Features **Hooks Component**: - Event-driven lifecycle observation at interaction/iteration/stage/model_call/tool_call boundaries - Built-in logging hook (structured JSON logs via slog) - Built-in tracing hook (OpenTelemetry spans with W3C trace context propagation) - Extensible registration API for custom hooks (metrics, audit, etc.) **Event Types**: - Lifecycle events: interaction/iteration/stage/model_call/tool_call × start/end - Error events: *.error emitted before *.end when operations fail - interaction.cancel for context cancellation - interaction.degrade for tool failures or fallback responses - State metadata: Error/Degraded/FallbackUsed fields provide additional context **ReAct Agent Integration**: - Hooks fire at every major lifecycle boundary - Trace context flows through interaction → iteration → stage → model/tool calls - Panic-safe: all exit paths (including panic recovery) drain agent channels to prevent goroutine leaks and ensure trace tail spans emit - Added regression test TestStreamChatDrainsChannelsOnPanic **Configuration**: - Component-based loading via hooks.yaml - Tracing supports grpc/http protocols, configurable sampling, content capture levels - Environment variable overrides (OTEL_EXPORTER_OTLP_ENDPOINT, etc.) ## Implementation Details - Trace IDs propagate via context; agent.Channels.SetTraceID enables correlation - Tracing hook defers content serialization until span recording to avoid overhead on unsampled traces - External hooks receive eagerly-snapshotted content (documented in Registration.CaptureContent) - Tool call failures set state.Degraded and emit tool_call.error before tool_call.end - context.Canceled mapped to interaction.cancel event ## Dependencies - Upgrade sonic to v1.15.2 for Go 1.26 compatibility - Add go.opentelemetry.io/otel/* packages for tracing ## Documentation - component/hooks/README.md: architecture, usage, custom hook guide - hooks.yaml: inline event reference for developers - Available events section in README ## Testing All tests pass: - go test ./component/hooks/... - go test ./component/agent/react/... - go test ./component/server/engine/... Addresses review feedback from PR apache#1536: - Error/cancel/degrade event types for metrics/audit hooks - lazyContent semantics clarified in docs - Panic drain regression test added - Event vocabulary documented in hooks.yaml and README - sonic compatibility issue resolved
chaojixinren
force-pushed
the
feat/hook-system
branch
from
August 23, 2026 04:41
054e4c5 to
1221da6
Compare
Add lifecycle hooks infrastructure and OpenTelemetry tracing integration for the ReAct agent, enabling observability via structured logging and distributed tracing. **Hooks Component**: - Event-driven lifecycle observation at interaction/iteration/stage/model_call/tool_call boundaries - Built-in logging hook (structured JSON logs via slog) - Built-in tracing hook (OpenTelemetry spans with W3C trace context propagation) - Extensible registration API for custom hooks (metrics, audit, etc.) **Event Types**: - Lifecycle events: interaction/iteration/stage/model_call/tool_call × start/end - Error events: *.error emitted before *.end when operations fail - interaction.cancel for context cancellation - interaction.degrade for tool failures or fallback responses - State metadata: Error/Degraded/FallbackUsed fields provide additional context **ReAct Agent Integration**: - Hooks fire at every major lifecycle boundary - Trace context flows through interaction → iteration → stage → model/tool calls - Panic-safe: all exit paths (including panic recovery) drain agent channels to prevent goroutine leaks and ensure trace tail spans emit - Added regression test TestStreamChatDrainsChannelsOnPanic **Configuration**: - Component-based loading via hooks.yaml - Tracing supports grpc/http protocols, configurable sampling, content capture levels - Environment variable overrides (OTEL_EXPORTER_OTLP_ENDPOINT, etc.) - Trace IDs propagate via context; agent.Channels.SetTraceID enables correlation - Tracing hook defers content serialization until span recording to avoid overhead on unsampled traces - External hooks receive eagerly-snapshotted content (documented in Registration.CaptureContent) - Tool call failures set state.Degraded and emit tool_call.error before tool_call.end - context.Canceled mapped to interaction.cancel event - Upgrade sonic to v1.15.2 for Go 1.26 compatibility - Add go.opentelemetry.io/otel/* packages for tracing - component/hooks/README.md: architecture, usage, custom hook guide - hooks.yaml: inline event reference for developers - Available events section in README All tests pass: - go test ./component/hooks/... - go test ./component/agent/react/... - go test ./component/server/engine/... Addresses review feedback from PR apache#1536: - Error/cancel/degrade event types for metrics/audit hooks - lazyContent semantics clarified in docs - Panic drain regression test added - Event vocabulary documented in hooks.yaml and README - sonic compatibility issue resolved
chaojixinren
force-pushed
the
feat/hook-system
branch
from
August 23, 2026 04:47
1221da6 to
0114c79
Compare
|
Author
感谢详细的 review！ 已修复：
说明：
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces a config-driven, telemetry-agnostic agent hooks system in
ai/, withlogging and OpenTelemetry (OTLP) tracing as the first observers. It instruments
the full ReAct lifecycle — interaction, iteration, stage, model call, and tool call —
without coupling the strategy code to any specific backend.
Motivation
Resolves #1525. There was previously no consistent way to observe agent execution or trace
a request end-to-end across the Agent ↔ MCP boundary, making agent behavior hard to debug
and spans impossible to correlate.
What changed
ai/component/hooks/): immutable lifecycle events(
interaction / iteration / stage / model_call / tool_call×start / end), read-onlyStatesnapshots, per-event registration, and panic isolation per hook.context.Contextfor nested work; exactly oneDerivesContextregistration is accepted (a Go context carries one span lineage — fan outvia a Collector instead). Contexts returned by plain observational hooks are ignored.
(
gen_ai.operation.name,gen_ai.request.model,gen_ai.provider.name,gen_ai.conversation.id,gen_ai.tool.name,gen_ai.tool.call.id,gen_ai.input/output.messages,gen_ai.usage.*,agent.fallback.*,error.type, …).Content is serialized lazily only when the span
IsRecording(), and content capture isopt-in (
capture_content: nonedefault;truncated≤ 4096 bytes;full).traceparent/tracestate/baggageare honored inbound,propagated to MCP HTTP calls, and the active trace ID is returned on SSE responses via
X-Trace-ID(CORS-exposed).FallbackReason), writtento both the model-call span and the stage span, with correct
Evidencetext; tool failuresare recorded as
error.type+agent.degradedwithout faking agen_ai.tool.call.result.context.Canceledpropagates cleanly (onlyDeadlineExceededis a timeout); SSE disconnect cancellation stays detached from the running interaction.
type: hookscomponent withlogging/tracingblocks, JSON schemavalidation, and standard OTel env vars for endpoint and credentials.
Design constraints
Agent execution data.
nonefor credential/PII safety; payloads are not serializedon the hot path unless a matching hook opts in.
"*"for all).Agent Hooks + OTel Tracing — Completed Test Checklist
1. Unit Tests
go test -count=1 ./...— all passed (hooks, Agent, server engine, MCP tools, runtime, etc.)2. Integration Tests
go test -tags=integration -count=1 ./...— all passed3. Race / Static Analysis
go test -race ./component/hooks/... ./component/agent/...— passedgo vet ./...— passed4. E2E: Jaeger (OTLP)
f6477cd5b8d5a5cce9ae07a0ad1d84705. E2E: Langfuse (Docker 4.11.0)
AGENT invoke_agent→GENERATION chat qwen-max→TOOL lookup_serviceevents_full: model input/output, 7/5/12 tokens, session ID, tool call IDfdc809bacdcdbf5b7cd7798ba8f7c1cb6. Performance Benchmarks (0 allocs)
BenchmarkDisabledHookFastPathBenchmarkEmptyManagerFastPathBenchmarkHookContentDisabledBenchmarkHookContentLoggingOnly7. Cleanup
7bbcab1TestMultiTurnConversationtherefore shows 0/14