Fix/SAO-259 operation trace ownership - #187
Conversation
pradystar
changed the title
fercor-cisco
left a comment
fercor-cisco
left a comment
There was a problem hiding this comment.
🤖 This review was generated by the Astra agent (claude-opus-4-8). It may contain mistakes.
Verdict: approve — Correctly implements the maintainer-approved ownership fix with solid tests; remaining points are behavior-change confirmations and minor gaps, not blockers.
General Comments
- 🟡 minor (design): Several handler defaults flip auto-flush off:
flush_on_chain_end(langchain base/async/middleware) now resolves to False unless aningestion_hookis set,flush_on_crew_completeddefaults to False, andflush_on_trace_enddefaults to False for the OpenAI processor. This is the intended consequence of relying on the BatchSpanProcessor's scheduled export (per the SAO-259 discussion), but it means short-lived processes that create a handler, emit a trace, and exit without callingterminate()/provider shutdown can now lose the final batch that aflush()used to deliver synchronously.atexit.register(self.terminate)mitigates this for normal interpreter exit, but not foros._exit, worker processes killed by a supervisor, or embedded runtimes. Worth a one-line note in the handler docs/changelog so users of ephemeral jobs know to callterminate()explicitly.
Follow-ups
Suggested follow-up work that could be tracked as Jira tickets:
src/splunk_ao/decorator.py:1126-1136: The sync/async generator wrappers now re-raise exceptions from the generator body (except BaseException: error = exc; raise) instead of the previous behavior of catchingExceptionand swallowing it with an error log. This makes generators consistent with the non-generator paths and is an improvement, but it is a behavior change for any caller that relied on decorated generators never propagating their internal errors. Worth calling out in the changelog.src/splunk_ao/decorator.py:884-894:_complete_callrecomputesoutputfromfinal_params/resultand_handle_call_resultindependently recomputes essentially the sameoutput. Consider computing it once and threading it through to avoid the two code paths drifting apart in future edits.
| except Exception: | ||
| if owned_trace is not None: | ||
| self._conclude_owned_trace(owned_trace, output="", status_code=500) | ||
| _logger.warning("Failed to commit handler telemetry", exc_info=True) |
There was a problem hiding this comment.
🟡 minor (design): commit() previously let exceptions from start_trace/log_node_tree/conclude/flush propagate (only the finally cleaned up). This new except Exception: block swallows all of them and downgrades to a warning. Silently dropping a partially-built trace is defensible for a telemetry callback, but it's a real change in error-propagation semantics for langchain/crewai integrations. Please confirm this swallowing is intended (and ideally note it) — if a framework relied on on_chain_end surfacing handler errors, it will now go quiet. Same pattern was added in base_async_handler.async_commit and openai_agents/handler.on_trace_end.
🤖 Generated by the Astra agent
There was a problem hiding this comment.
Confirmed intentional. These are telemetry callback failures and must not fail the instrumented LangChain/CrewAI operation. The handler logs the failure, cleans up only a trace it owns, and preserves any caller-owned trace.
| def _conclude_current_trace_on_failure(self) -> None: | ||
| if self._owned_trace is None: | ||
| return | ||
|
|
||
| current_parent = self._splunk_ao_logger.current_parent() | ||
| if current_parent is None: | ||
| return | ||
|
|
||
| root = current_parent | ||
| while root._parent is not None: | ||
| root = root._parent | ||
| if root is self._owned_trace: | ||
| self._splunk_ao_logger.conclude(output="", status_code=500, conclude_all=True) |
There was a problem hiding this comment.
🟡 minor (testing): _conclude_current_trace_on_failure() is new failure-path logic (root-walk + guarded conclude(conclude_all=True)), but unlike the base/async handlers — which got test_commit_failure_* tests — there's no test exercising a _commit_trace failure for the OpenAI processor. Add a test that patches _commit_trace/_log_node_tree to raise and asserts the owned trace is concluded (and a caller-owned trace is left intact), matching test_commit_failure_concludes_only_handler_owned_trace / test_commit_failure_preserves_caller_owned_trace.
🤖 Generated by the Astra agent
| try: | ||
| result = await func(*args, **kwargs) | ||
| except Exception: | ||
| except BaseException as exc: | ||
| _logger.error("Error while executing function in async_wrapper", exc_info=True) | ||
| if call_state is not None: | ||
| self._finalize_call(span_type, span_params, None, call_state, error=exc) | ||
| raise |
There was a problem hiding this comment.
🟡 minor (design): Widening the catch to BaseException means that on CancelledError/KeyboardInterrupt/SystemExit the wrapper now runs full finalization (_finalize_call → _handle_call_result → logger.conclude) before re-raising. That's good for trace correctness, but it does non-trivial work on the cancellation/shutdown path. Since conclude/emit only touch the batch sink here it's acceptable, just flagging that a slow or blocking sink operation would now delay cancellation. Consider keeping finalization minimal on BaseException (e.g. conclude with a fixed status and skip optional serialization) if you want cancellation to stay snappy.
🤖 Generated by the Astra agent
There was a problem hiding this comment.
Confirmed intentional. This path performs bounded local telemetry finalization and enqueues the completed span; it does not force-flush or perform network delivery. The original CancelledError, KeyboardInterrupt, or SystemExit is then immediately re-raised.
# Conflicts: # src/splunk_ao/decorator.py # tests/test_decorator.py
|
2 participants
Summary
Fixes operation trace ownership so independent top-level SDK calls produce separate OTel traces without
relying on flush() as a trace boundary.
Why
Top-level @log calls could leave their internal trace envelope active, causing later calls using the
singleton logger to reuse the same OTel trace. Handler auto-flushing also reduced the effectiveness of
BatchSpanProcessor batching.
What changed
Tracks per-call trace ownership across synchronous, asynchronous, exception, and generator paths.
Concludes only decorator- or handler-owned traces while preserving nested, manually owned, and distributed
upstream contexts.
Preserves OpenAI tool-call trace lifecycle behavior.
Makes flush() a drain-only operation that does not end active operations or clear trace context.
Lets normal handlers rely on scheduled OTel batch export while retaining explicit force-flush support and
legacy ingestion-hook compatibility.
Sanitizes reserved Splunk AO routing attributes from generic OTel Resource configuration and applies only
authoritative SDK routing.
Preserves service.name and other non-routing Resource attributes.
Testing
OTel trace IDs (3/3), with no active trace or parent left after eac