feat(observability): end-to-end task attribution and cross-plane trace correlation

[krokoko]

Component

API or orchestration, Agent (Python runtime), CDK / infrastructure

Describe the feature

Implement end-to-end task attribution and unified cross-plane trace correlation so every action in a task's lifecycle can be traced back to {task_id, user_id, repo} and a single shared trace root — across the orchestrator, agent runtime, GitHub/API calls, TaskEvents, OpenTelemetry spans, and S3 trace artifacts.

Today attribution is partial: per-session IAM scoping (#209, shipped) tags tenant-data access; git commits carry Task-Id trailers; TaskEvents and task records hold operational fields. Gaps remain in orchestrator structured logs, agent OTel baggage propagation, external API call correlation, and stitching MicroVM traces to platform-side events for incident response.

This issue is the platform identity contract — distinct from Bedrock billing attribution (#215) and compliance export schema (#237), though all three should share the same correlation field names.

Use case

• Incident response: Answer "who triggered this, on which repo, in which task?" without manually joining CloudWatch, DDB, and S3.
• Audit and compliance: Feed RFC: Governance planes — analytics and compliance export #237's abca.audit.v1 correlation block from real propagated context, not ad-hoc assembly.
• Cost and ops debugging: Join Bedrock invocation logs (feat: Bedrock cost attribution — session tags, request metadata, and operator FinOps guidance #215 metadata) to orchestrator and agent spans via shared trace_id / task_id.
• Vision tenet 7: Observable, attributable, replayable — the anti-goal "autonomy without attribution" must not apply to shipped paths.

Proposed solution

Phase 0 — Correlation contract

• Document a stable correlation envelope: { task_id, user_id, repo, trace_id, session_id? } and where each field is authoritative.
• Define naming consistency across orchestrator logs, agent OTel resource/span attributes, TaskEvents optional fields, and git/API metadata.

Phase 1 — Propagation MVP

• Orchestrator: inject correlation into structured logs and durable-step context for every task transition.
• Agent: propagate envelope via OTel baggage / span attributes from session start through pipeline phases and progress events.
• Ensure progress_writer and terminal events include correlation stamps where missing.

Phase 2 — Cross-plane trace root

• Single trace root per task linking orchestrator spans, MicroVM agent spans, and (where available) S3 trace artifact URI.
• Gap-fill beyond existing AgentCore session baggage (OBSERVABILITY.md); align with bgagent trace download artifact paths.

Phase 3 — Downstream attribution (follow-up OK in same issue or child)

• GitHub commits/PR comments: consistent actor + task metadata (extend existing trailers).
• External webhook/API calls from handlers: correlation headers or structured log fields.

Out of scope for initial delivery

• Delegation chain propagation (cryptographically signed user_id → orchestrator → agent chain) — separate follow-up once basic propagation is stable.
• Replacing in-app cost_usd or FinOps dashboards (feat: Bedrock cost attribution — session tags, request metadata, and operator FinOps guidance #215, RFC: Governance planes — analytics and compliance export #237).

Acceptance criteria

• Correlation contract documented in design docs (source + Starlight sync).
• { task_id, user_id, repo, trace_id } present in orchestrator structured logs for admission → terminal transitions.
• Agent OTel spans carry the same correlation attributes; baggage survives between pipeline phases.
• TaskEvents written during a run are joinable to orchestrator logs and trace artifacts by task_id and trace_id.
• Tests: unit/integration for context injection; manual or integ verification documented (pairs with feat(cdk): CDK integ-tests for deployed runtime E2E verification #236).
• No regression to feat(security): per-session IAM scoping — session-tagged AssumeRole, DynamoDB leading-key conditions, Bedrock ARN allowlist #209 SessionRole tenant isolation.

Other information

• Roadmap: End-to-end task attribution, Unified cross-plane trace correlation (docs/guides/ROADMAP.md).
• Related: feat(security): per-session IAM scoping — session-tagged AssumeRole, DynamoDB leading-key conditions, Bedrock ARN allowlist #209 (per-session IAM), feat: Bedrock cost attribution — session tags, request metadata, and operator FinOps guidance #215 (Bedrock cost attribution), RFC: Governance planes — analytics and compliance export #237 (governance/compliance export), feat(cdk): CDK integ-tests for deployed runtime E2E verification #236 (deployed E2E verification).
• Priority context: Prerequisite for trustworthy cost chargeback and compliance exports; sequenced after or in parallel with feat(cdk): CDK integ-tests for deployed runtime E2E verification #236 smoke integ.

Acknowledgements

• I may be able to implement this feature
• This might be a breaking change

[theagenticguy]

The deferred Delegation chain propagation (cryptographically signed user_id → orchestrator → agent) called out as out-of-scope here is now spec'd in #249 — identity propagation on the credential plane, complementing this issue's data plane. #249 reuses this issue's {task_id, user_id, repo, trace_id} envelope as the token binding key so log attribution and token binding never diverge. Drafted by Bonk, filed by Laith.

[krokoko]

Roadmap: unified cross-plane trace correlation (explicit AC)

Ensure scope includes:

• Single trace root per task across orchestrator, MicroVM OpenTelemetry, TaskEvents, and S3 trace artifacts
• Gap-fill beyond existing AgentCore session baggage (docs/design/OBSERVABILITY.md)

[krokoko]

Suggested sub-deliverable: Claude Code native OTel events + operator runbook

The harness already wires Claude Code CLI telemetry behind ENABLE_CLI_TELEMETRY=1 in agent/src/runner.py (CLAUDE_CODE_ENABLE_TELEMETRY, OTLP logs export, OTEL_RESOURCE_ATTRIBUTES with task.id / repo.url / agent.model). CDK leaves it disabled today (ENABLE_CLI_TELEMETRY commented in cdk/src/stacks/agent.ts) pending verification that the ADOT sidecar forwards OTLP logs, not just traces.

Recommend folding the following into this issue (or a child task) rather than a separate feature request:

Deliverables

1. Enable + verify — Turn on ENABLE_CLI_TELEMETRY in the AgentCore runtime env after confirming claude_code.api_request / claude_code.tool_result / claude_code.tool_decision records land in CloudWatch (application log group or AgentCore runtime log stream).
2. Session correlation — Ensure task.id on OTEL_RESOURCE_ATTRIBUTES joins to the correlation envelope defined in Phase 0 of this issue.
3. Operator runbook — Add docs/guides/CLAUDE_CODE_TELEMETRY.md (sync via Starlight) with:
   • Event type reference (claude_code.api_request carries cost_usd, input_tokens, output_tokens, duration_ms)
   • Copy-paste CloudWatch Logs Insights queries: per-task cost rollup, tool histogram, PreToolUse / hook blocks via claude_code.tool_decision
   • Known limitations: Claude Code emits OTLP events (LogRecords), not spans — Traces tab may be empty while Events/logs are populated; subagent attribution gaps
4. Docs cross-links — From docs/design/OBSERVABILITY.md and docs/guides/COST_ATTRIBUTION.md (complements feat: Bedrock cost attribution — session tags, request metadata, and operator FinOps guidance #215 Bedrock billing tags).
5. Optional — Dashboard widgets for cost_usd by task.id when attributes are present.

Why under #245

This is the per-model-call granularity layer for the same attribution story: custom harness spans + TaskEvents give task-level correlation; Claude Code events add prompt/tool-level cost and token forensics without parsing unstructured logs.

Acceptance sketch

• One dry-run task produces queryable claude_code.api_request records tagged with task.id
• Runbook queries documented and validated against a real log group
• ENABLE_CLI_TELEMETRY default documented (on vs opt-in) with sidecar forwarding notes