Skip to content

fix(tracing): capture span body exceptions and export SGP status=ERROR#460

Open
NiteshDhanpal wants to merge 28 commits into
nextfrom
fix/tracing-capture-span-error-status
Open

fix(tracing): capture span body exceptions and export SGP status=ERROR#460
NiteshDhanpal wants to merge 28 commits into
nextfrom
fix/tracing-capture-span-error-status

Conversation

@NiteshDhanpal

@NiteshDhanpal NiteshDhanpal commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Problem

Spans exported to SGP always show status=SUCCESS, even when the operation they represent failed.

Root cause, in three parts:

  • The SGP span (scale_gp_beta) defaults status="SUCCESS" and only flips to "ERROR" inside its own __exit__ context manager. The agentex processor builds SGP spans via create_span(...) and flushes them directly, so __exit__ never runs.
  • The agentex Span type (generated from the OpenAPI spec) has no status/error field.
  • Trace.span() / AsyncTrace.span() ended spans in a bare finally, so a body exception was never recorded before the span was flushed.

Net effect: a failing operation is indistinguishable from a successful one in SGP.

Fix (SDK-only, no schema/backend change)

Capture → carry → map:

  1. Capture — both context managers now except Exception as exc: set_span_error(span, exc); raise.
  2. Carryspan_error.py stores the failure under the reserved span.data["__error__"] key, following the existing __span_type__/__source__ convention. data is a real field, so it survives model_copy(deep=True) and round-trips to both the SGP and agentex-native stores. (A first-class status/error field would require the OpenAPI → Stainless → backend → migration chain; this achieves the same SGP outcome without it.)
  3. Map_build_sgp_span() sets sgp_span.status = "ERROR" + error/error.type/error.message metadata when an error is present, matching SGP's native __exit__ shape.

Exceptions still propagate unchanged. asyncio.CancelledError / KeyboardInterrupt are intentionally not flagged (control flow, not failures) — broaden the except to BaseException if that changes.

Tests

tests/lib/core/tracing/test_span_error.py:

  • set_span_error/get_span_error helpers (incl. list-shaped data no-op).
  • Sync + async context managers record the error and re-raise; success path stays clean.
  • _build_sgp_span maps error → status=ERROR + metadata; no-error path stays SUCCESS.

ruff check clean; new + existing tracing tests pass (40 passed).

🤖 Generated with Claude Code

Greptile Summary

This PR fixes silent status=SUCCESS on failing SGP spans by wiring exception capture into both Trace.span() / AsyncTrace.span() context managers and threading the captured error through to the SGP span builder.

  • Capture: both sync and async context managers now except Exception as exc, call set_span_error(span, exc), and re-raise; non-Exception subclasses (asyncio.CancelledError, KeyboardInterrupt) intentionally bypass the handler.
  • Carry: span_error.py stores the failure as span.data[\"__error__\"] = {\"type\": ..., \"message\": ...}, following the existing __span_type__/__source__ double-underscore convention; the value survives recursive_model_dump and model_copy(deep=True) since those preserve all dict keys.
  • Map: _build_sgp_span calls sgp_span.set_error(error_type=..., error_message=...) when an error is present, flipping SGP status to \"ERROR\" and populating error metadata, matching what SGP's own __exit__ produces.

Confidence Score: 5/5

The change is safe to merge — it adds exception capture to both context managers and a minimal mapper in the SGP span builder, with no mutations to the processor pipeline, API contracts, or span flushing logic.

The three-part fix (capture → carry → map) is internally consistent: set_span_error writes a plain-dict value under a reserved key that survives recursive_model_dump (verified by reading model_utils.py) and model_copy(deep=True); the except/raise in both context managers correctly fires before the finally end_span call; and the SGP builder reads the error before returning the span. No pre-existing behavior is altered on the success path.

No files require special attention.

Important Files Changed

Filename Overview
src/agentex/lib/core/tracing/span_error.py New module providing set_span_error/get_span_error helpers; correctly handles None/dict/list-shaped span.data and follows the existing double-underscore reserved-key convention.
src/agentex/lib/core/tracing/trace.py Adds except/raise around yield in both sync and async span() context managers; only Exception is caught (intentionally excludes CancelledError/KeyboardInterrupt), and the finally block always calls end_span, preserving existing behavior.
src/agentex/lib/core/tracing/processors/sgp_tracing_processor.py _build_sgp_span now reads get_span_error and calls set_error on the SGPSpan when an error is present; the change is minimal, correctly ordered after start_time assignment, and does not affect the no-error path.
tests/lib/core/tracing/test_span_error.py Covers helpers, sync/async context-manager capture, and _build_sgp_span mapping via a FakeSGPSpan; tests patch create_span so the real SGPSpan.set_error signature is not verified but the SDK contract is described accurately in the PR.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Caller
    participant Trace as Trace / AsyncTrace
    participant SpanError as span_error.py
    participant Processor as SGP Processor
    participant SGPSpan as SGPSpan (_build_sgp_span)

    Caller->>Trace: async with trace.span("op") as span
    Trace->>Trace: start_span() creates Span
    Trace-->>Caller: yields span

    alt Body raises Exception
        Caller->>Trace: raises Exception
        Trace->>SpanError: set_span_error(span, exc)
        SpanError->>SpanError: "span.data[__error__] = {type, message}"
        Trace->>Trace: re-raise
    end

    Note over Trace: finally block always runs
    Trace->>Trace: end_span(span)
    Trace->>Trace: recursive_model_dump preserves __error__
    Trace->>Processor: "on_span_end(span.model_copy(deep=True))"
    Processor->>SGPSpan: _build_sgp_span(span, env_vars)
    SGPSpan->>SpanError: get_span_error(span)
    SpanError-->>SGPSpan: dict or None

    alt Error present
        SGPSpan->>SGPSpan: set_error(error_type, error_message)
        Note over SGPSpan: status = ERROR
    else No error
        Note over SGPSpan: status remains SUCCESS
    end

    SGPSpan-->>Processor: SGPSpan with correct status
    Processor->>Processor: flush / upsert_batch
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Caller
    participant Trace as Trace / AsyncTrace
    participant SpanError as span_error.py
    participant Processor as SGP Processor
    participant SGPSpan as SGPSpan (_build_sgp_span)

    Caller->>Trace: async with trace.span("op") as span
    Trace->>Trace: start_span() creates Span
    Trace-->>Caller: yields span

    alt Body raises Exception
        Caller->>Trace: raises Exception
        Trace->>SpanError: set_span_error(span, exc)
        SpanError->>SpanError: "span.data[__error__] = {type, message}"
        Trace->>Trace: re-raise
    end

    Note over Trace: finally block always runs
    Trace->>Trace: end_span(span)
    Trace->>Trace: recursive_model_dump preserves __error__
    Trace->>Processor: "on_span_end(span.model_copy(deep=True))"
    Processor->>SGPSpan: _build_sgp_span(span, env_vars)
    SGPSpan->>SpanError: get_span_error(span)
    SpanError-->>SGPSpan: dict or None

    alt Error present
        SGPSpan->>SGPSpan: set_error(error_type, error_message)
        Note over SGPSpan: status = ERROR
    else No error
        Note over SGPSpan: status remains SUCCESS
    end

    SGPSpan-->>Processor: SGPSpan with correct status
    Processor->>Processor: flush / upsert_batch
Loading

Reviews (2): Last reviewed commit: "fix(tracing): capture span body exceptio..." | Re-trigger Greptile

declan-scale and others added 27 commits June 22, 2026 15:59
…gModel (#355)

Co-authored-by: stainless-app[bot] <142633134+stainless-app[bot]@users.noreply.github.com>
Co-authored-by: Declan Brady <declan.brady@scale.com>
Co-authored-by: Nitesh Dhanpal <NiteshDhanpal@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…est) (#438)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…5f2fe01d48613aaa6fcf--merge-conflict' into next
add stable schedule handle endpoints
@smoreinis

Copy link
Copy Markdown
Contributor

can we target next instead of main here and also fix the diff to pass lint?

Spans exported to SGP always showed status=SUCCESS even when the operation
they represent failed. The SGP span defaults status to "SUCCESS" and only
flips to "ERROR" inside its own __exit__ context manager, but the agentex
processor builds SGP spans via create_span(...) and flushes them directly,
so __exit__ never runs. The Trace.span()/AsyncTrace.span() context managers
also ended spans in a bare finally, so a body exception was never recorded.

Capture the exception in both context managers and carry it on the span so
the SGP processor can map it:

- add span_error.py with set_span_error/get_span_error, storing the failure
  under the reserved span.data["__error__"] key (the Span model is generated
  from the OpenAPI spec and has no status/error field; data is a real field
  that survives model_copy(deep=True) and round-trips to both stores)
- Trace.span()/AsyncTrace.span(): except -> set_span_error(span, exc); raise
- _build_sgp_span(): when an error is present, set sgp_span.status = "ERROR"
  plus error/error.type/error.message metadata (matching SGP's native
  __exit__ shape)

Exceptions still propagate; asyncio.CancelledError/KeyboardInterrupt are not
flagged (control flow, not failures).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NiteshDhanpal NiteshDhanpal force-pushed the fix/tracing-capture-span-error-status branch from a2fa8d5 to 76af59f Compare July 10, 2026 17:00
@NiteshDhanpal NiteshDhanpal changed the base branch from main to next July 10, 2026 17:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants