fix(tracing): capture span body exceptions and export SGP status=ERROR#460
Open
NiteshDhanpal wants to merge 28 commits into
Open
fix(tracing): capture span body exceptions and export SGP status=ERROR#460NiteshDhanpal wants to merge 28 commits into
NiteshDhanpal wants to merge 28 commits into
Conversation
… delivery adapters, emitter) (#412)
…gModel (#355) Co-authored-by: stainless-app[bot] <142633134+stainless-app[bot]@users.noreply.github.com> Co-authored-by: Declan Brady <declan.brady@scale.com> Co-authored-by: Nitesh Dhanpal <NiteshDhanpal@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…quivalence [AGX1-373] (#414)
…est) (#438) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…5f2fe01d48613aaa6fcf--merge-conflict' into next
…e/list in schedules
add stable schedule handle endpoints
Contributor
|
can we target next instead of main here and also fix the diff to pass lint? |
Spans exported to SGP always showed status=SUCCESS even when the operation they represent failed. The SGP span defaults status to "SUCCESS" and only flips to "ERROR" inside its own __exit__ context manager, but the agentex processor builds SGP spans via create_span(...) and flushes them directly, so __exit__ never runs. The Trace.span()/AsyncTrace.span() context managers also ended spans in a bare finally, so a body exception was never recorded. Capture the exception in both context managers and carry it on the span so the SGP processor can map it: - add span_error.py with set_span_error/get_span_error, storing the failure under the reserved span.data["__error__"] key (the Span model is generated from the OpenAPI spec and has no status/error field; data is a real field that survives model_copy(deep=True) and round-trips to both stores) - Trace.span()/AsyncTrace.span(): except -> set_span_error(span, exc); raise - _build_sgp_span(): when an error is present, set sgp_span.status = "ERROR" plus error/error.type/error.message metadata (matching SGP's native __exit__ shape) Exceptions still propagate; asyncio.CancelledError/KeyboardInterrupt are not flagged (control flow, not failures). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
a2fa8d5 to
76af59f
Compare
declan-scale
approved these changes
Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Spans exported to SGP always show
status=SUCCESS, even when the operation they represent failed.Root cause, in three parts:
scale_gp_beta) defaultsstatus="SUCCESS"and only flips to"ERROR"inside its own__exit__context manager. The agentex processor builds SGP spans viacreate_span(...)and flushes them directly, so__exit__never runs.Spantype (generated from the OpenAPI spec) has nostatus/errorfield.Trace.span()/AsyncTrace.span()ended spans in a barefinally, so a body exception was never recorded before the span was flushed.Net effect: a failing operation is indistinguishable from a successful one in SGP.
Fix (SDK-only, no schema/backend change)
Capture → carry → map:
except Exception as exc: set_span_error(span, exc); raise.span_error.pystores the failure under the reservedspan.data["__error__"]key, following the existing__span_type__/__source__convention.datais a real field, so it survivesmodel_copy(deep=True)and round-trips to both the SGP and agentex-native stores. (A first-classstatus/errorfield would require the OpenAPI → Stainless → backend → migration chain; this achieves the same SGP outcome without it.)_build_sgp_span()setssgp_span.status = "ERROR"+error/error.type/error.messagemetadata when an error is present, matching SGP's native__exit__shape.Exceptions still propagate unchanged.
asyncio.CancelledError/KeyboardInterruptare intentionally not flagged (control flow, not failures) — broaden theexcepttoBaseExceptionif that changes.Tests
tests/lib/core/tracing/test_span_error.py:set_span_error/get_span_errorhelpers (incl. list-shapeddatano-op)._build_sgp_spanmaps error →status=ERROR+ metadata; no-error path staysSUCCESS.ruff checkclean; new + existing tracing tests pass (40 passed).🤖 Generated with Claude Code
Greptile Summary
This PR fixes silent
status=SUCCESSon failing SGP spans by wiring exception capture into bothTrace.span()/AsyncTrace.span()context managers and threading the captured error through to the SGP span builder.except Exception as exc, callset_span_error(span, exc), and re-raise; non-Exceptionsubclasses (asyncio.CancelledError,KeyboardInterrupt) intentionally bypass the handler.span_error.pystores the failure asspan.data[\"__error__\"] = {\"type\": ..., \"message\": ...}, following the existing__span_type__/__source__double-underscore convention; the value survivesrecursive_model_dumpandmodel_copy(deep=True)since those preserve all dict keys._build_sgp_spancallssgp_span.set_error(error_type=..., error_message=...)when an error is present, flipping SGP status to\"ERROR\"and populating error metadata, matching what SGP's own__exit__produces.Confidence Score: 5/5
The change is safe to merge — it adds exception capture to both context managers and a minimal mapper in the SGP span builder, with no mutations to the processor pipeline, API contracts, or span flushing logic.
The three-part fix (capture → carry → map) is internally consistent: set_span_error writes a plain-dict value under a reserved key that survives recursive_model_dump (verified by reading model_utils.py) and model_copy(deep=True); the except/raise in both context managers correctly fires before the finally end_span call; and the SGP builder reads the error before returning the span. No pre-existing behavior is altered on the success path.
No files require special attention.
Important Files Changed
Sequence Diagram
%%{init: {'theme': 'neutral'}}%% sequenceDiagram participant Caller participant Trace as Trace / AsyncTrace participant SpanError as span_error.py participant Processor as SGP Processor participant SGPSpan as SGPSpan (_build_sgp_span) Caller->>Trace: async with trace.span("op") as span Trace->>Trace: start_span() creates Span Trace-->>Caller: yields span alt Body raises Exception Caller->>Trace: raises Exception Trace->>SpanError: set_span_error(span, exc) SpanError->>SpanError: "span.data[__error__] = {type, message}" Trace->>Trace: re-raise end Note over Trace: finally block always runs Trace->>Trace: end_span(span) Trace->>Trace: recursive_model_dump preserves __error__ Trace->>Processor: "on_span_end(span.model_copy(deep=True))" Processor->>SGPSpan: _build_sgp_span(span, env_vars) SGPSpan->>SpanError: get_span_error(span) SpanError-->>SGPSpan: dict or None alt Error present SGPSpan->>SGPSpan: set_error(error_type, error_message) Note over SGPSpan: status = ERROR else No error Note over SGPSpan: status remains SUCCESS end SGPSpan-->>Processor: SGPSpan with correct status Processor->>Processor: flush / upsert_batch%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%% sequenceDiagram participant Caller participant Trace as Trace / AsyncTrace participant SpanError as span_error.py participant Processor as SGP Processor participant SGPSpan as SGPSpan (_build_sgp_span) Caller->>Trace: async with trace.span("op") as span Trace->>Trace: start_span() creates Span Trace-->>Caller: yields span alt Body raises Exception Caller->>Trace: raises Exception Trace->>SpanError: set_span_error(span, exc) SpanError->>SpanError: "span.data[__error__] = {type, message}" Trace->>Trace: re-raise end Note over Trace: finally block always runs Trace->>Trace: end_span(span) Trace->>Trace: recursive_model_dump preserves __error__ Trace->>Processor: "on_span_end(span.model_copy(deep=True))" Processor->>SGPSpan: _build_sgp_span(span, env_vars) SGPSpan->>SpanError: get_span_error(span) SpanError-->>SGPSpan: dict or None alt Error present SGPSpan->>SGPSpan: set_error(error_type, error_message) Note over SGPSpan: status = ERROR else No error Note over SGPSpan: status remains SUCCESS end SGPSpan-->>Processor: SGPSpan with correct status Processor->>Processor: flush / upsert_batchReviews (2): Last reviewed commit: "fix(tracing): capture span body exceptio..." | Re-trigger Greptile