Skip to content

Fail closed on span customizer errors without dropping spans - #915

Draft
Andrew Kent (realark) wants to merge 1 commit into
mainfrom
ark/SDK-472-customizer-error-handling
Draft

Andrew Kent (realark) wants to merge 1 commit into
mainfrom
ark/SDK-472-customizer-error-handling

Conversation

@realark

@realark Andrew Kent (realark) commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

AI Summary

Implements only the span customizer hook-failure contract (SDK-472).

  • Stop the hook chain on invalid results or any BaseException except KeyboardInterrupt/SystemExit; export original identity/routing/protocol fields and original start/end timestamps plus context={"customizer_error": true}, never the original payload or earlier mutations.
  • Keep stripped records memoized across lazy-resolution and transport retries. Do not invent missing timestamps; later incremental records are independent. Preserve successful customization and dict-subclass conversion.
  • Log at most one private-data-free hook-failure error per process, including concurrent failures and registration changes.
  • Prevent failed-record attachment uploads and exempt SDK-marked stripped records from masking. Stripped records are merge/request barriers, so neighboring incremental content cannot contaminate cached failures. Send barrier batches in order to preserve the existing duplicate-row transaction protocol; successful-only batching is unchanged.
  • Update public API documentation and replace dropped-record tests with stripping coverage.

Verification

  • cd py && mise exec -- uv run nox -s test_core: 987 passed, 73 skipped, 12 xfailed (Python 3.12.12).
  • Core suite includes a real localhost HTTP smoke through the SDK HTTP connection and background exporter: success/failure/success on one span, HTTP 503 retry with identical stripped bytes, arbitrary masking, untouched cached failure, failed attachment suppression, and healthy attachment upload.
  • Regression coverage includes invalid returns/coroutines, dict subclasses, BaseException/GeneratorExit/CancelledError, process-control propagation, original mutated identity/timing, full content stripping, no-duration incremental updates, ordering, mixed batches, retry caching, local experiment cache, concurrent once-only diagnostics, privacy, and disabled error logging.
  • Ruff formatting/checks on the four changed files passed. Commit hooks passed: EOF, whitespace, Ruff format/check, codespell.

Limitations

  • Native Python export path only; OTel customizer support remains intentionally unchanged.
  • No remote Braintrust service or optional-provider integration matrix was exercised; HTTP smoke uses a local server and attachment upload behavior is observed with spies.
  • Only Python 3.12.12 was exercised locally; CI covers the remaining supported Python versions.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant