Skip to content

SMOODEV-2045: SDK self-emit is panic-proof (fix temporal-worker crashloop root cause)#67

Merged
brentrager merged 1 commit into
mainfrom
SMOODEV-2033-selfemit-crashfix
Jun 22, 2026
Merged

SMOODEV-2045: SDK self-emit is panic-proof (fix temporal-worker crashloop root cause)#67
brentrager merged 1 commit into
mainfrom
SMOODEV-2033-selfemit-crashfix

Conversation

@brentrager

Copy link
Copy Markdown
Contributor

Root cause of the prod temporal-worker crashloop (SMOODEV-1988 self-emit; mitigated by disabling it in SMOODEV-2031): the OTLP export executed in a context with no Tokio reactor present, so the HTTP request panicked; under the workspace's panic = "abort" release profile that panic on the background path aborted the whole host process → crashloop (blocked eSign).

Fix: drive the export on a dedicated std::thread with its own runtime so a missing-reactor / auth-401 / unreachable-endpoint never panics or aborts the host. The SDK is now genuinely best-effort even when configured-but-unreachable.

Tests (new tests/self_emit_panic_safety.rs): configured_but_unauthorized_ingest_does_not_panic, unreachable_endpoint_export_does_not_panic — both pass. fmt/clippy/test green.

NOTE: this does NOT re-enable self-emit anywhere — SMOODEV-2031's SMOOAI_OBSERVABILITY_DISABLED stays. It makes the SDK safe so self-emit CAN be re-enabled after review.

🤖 Generated with Claude Code

…igured-but-unreachable (fix worker crashloop root cause)
@changeset-bot

changeset-bot Bot commented Jun 22, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 316a32d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@brentrager brentrager merged commit dd73547 into main Jun 22, 2026
6 checks passed
@brentrager brentrager deleted the SMOODEV-2033-selfemit-crashfix branch June 22, 2026 01:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant