fix(telemetry): only shut down OTel providers the framework created - #6931
fix(telemetry): only shut down OTel providers the framework created#6931sofyat wants to merge 1 commit into
Conversation
`_shutdown_telemetry()` runs once per job from `JobContext._on_cleanup`, but
it shut down whatever tracer/logger/meter provider it found on the OTel
globals — including a provider supplied by the integrator.
Worker processes are reused across jobs, and an integrator's provider is
configured once at process start, so from the second job onward their
exporters were already dead. This affects the Langfuse setup in the tracing
docs (`docs.livekit.io/deploy/observability/tracing`), which passes a plain
`opentelemetry.sdk.trace.TracerProvider` to `set_tracer_provider`, as well as
API-level wrappers such as Logfire and dd-trace.
`_setup_cloud_tracer` now records what it creates (`_FrameworkTelemetry`), and
teardown follows ownership:
* providers the framework created are shut down in full, as before;
* for an adopted provider, only the processors the framework attached to it
are shut down. LiveKit Cloud's batch exporter is still flushed at job end,
while the integrator's pipeline keeps running.
The registry is also emptied on teardown, so a provider is never shut down
twice across jobs.
Root-logger cleanup is narrowed to the framework's own
`_TraceLevelLoggingHandler` for the same reason: the previous `LoggingHandler`
match would detach an integrator's OTel handler too.
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| tracer_provider.add_span_processor(_MetadataSpanProcessor(session_metadata)) | ||
| tracer_provider.add_span_processor(BatchSpanProcessor(span_exporter)) | ||
| span_processor = BatchSpanProcessor(span_exporter) | ||
| tracer_provider.add_span_processor(span_processor) | ||
| if not owns_tracer_provider: | ||
| # the provider outlives the job; shut down just our exporter. | ||
| _framework_telemetry.add_processor(span_processor) |
There was a problem hiding this comment.
🟡 Integrator's own traces and logs get stamped with a finished job's room and job identifiers
The previous job's metadata-tagging step is left attached to an integrator-supplied tracing/logging pipeline (add_span_processor(_MetadataSpanProcessor(...)) at livekit-agents/livekit/agents/telemetry/traces.py:413) instead of being removed when the job ends, so the integrator's own traces and logs keep being stamped with a finished job's room and job identifiers.
Impact: On reused worker processes, an integrator's telemetry becomes polluted with stale, incorrect job/room metadata, and per-job tagging overhead grows without bound.
Why the tagging step outlives the job on adopted providers
Before this PR, _shutdown_telemetry called shutdown() on the whole adopted provider, so nothing produced through it survived. After this PR the adopted provider is deliberately kept alive and only the framework's BatchSpanProcessor / BatchLogRecordProcessor are shut down (_framework_telemetry.add_processor(...) at livekit-agents/livekit/agents/telemetry/traces.py:416-418 and :439-441).
But _MetadataSpanProcessor(session_metadata) (:413) and _MetadataLogProcessor(session_metadata) (:436) are neither registered for shutdown nor otherwise detached. _MetadataSpanProcessor.on_start (:204-205) applies session_metadata (which contains room_id/job_id for that specific job) to every span the adopted provider starts. Since the provider now lives on across jobs, the integrator's own application spans/logs created after the job ends get tagged with that job's room_id/job_id. The next job's _setup_cloud_tracer appends yet another _MetadataSpanProcessor, so these tagging processors accumulate one pair per job for the lifetime of the reused worker process, each running on every span/log.
The author acknowledges this as "Processor accumulation" in the PR description, but it is a direct, real consequence of the ownership change on exactly the integrator path the PR targets.
Prompt for agents
On the adopted-provider path in _setup_cloud_tracer (livekit-agents/livekit/agents/telemetry/traces.py around lines 412-418 for spans and 436-441 for logs), the framework attaches a _MetadataSpanProcessor / _MetadataLogProcessor carrying this job's session_metadata (room_id, job_id, etc.) to a provider it does NOT own. Since the fix now leaves adopted providers alive across jobs, these metadata processors are never detached, so (1) the integrator's own spans/logs get stamped with a finished job's room_id/job_id, and (2) a new pair accumulates every job for the process lifetime.
Consider tracking these metadata processors alongside the batch processor and detaching them at job teardown. There is no public remove_span_processor API on TracerProvider, so options include: making the metadata processors mutable/no-op-able (e.g. clearing their metadata on shutdown so on_start becomes a no-op), or having _FrameworkTelemetry hold references and disabling them in take()/shutdown. Ensure the batch exporter still flushes at job end while the integrator's own pipeline keeps running with correct (or no framework) metadata.
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
Problem
_shutdown_telemetry()runs once per job fromJobContext._on_cleanup, but it shut down whichever tracer / logger / meter provider it found on the OTel globals — including a provider supplied by the integrator.Worker processes are reused across jobs, and an integrator's provider is configured once at process start, so from the second job onward their exporters were already dead.
This is reachable from the setup in the tracing docs: the Langfuse example passes a plain
opentelemetry.sdk.trace.TracerProvidertoset_tracer_provider,_setup_cloud_traceradopts it, and_shutdown_telemetrythen callsshutdown()on it at the end of every job. It also affects providers that wrap at the OTel API level, such as Logfire and dd-trace.Change
_setup_cloud_tracernow records what it creates (_FrameworkTelemetry), and teardown follows ownership:The registry is emptied on teardown, so a provider is never shut down twice across jobs.
Root-logger cleanup is narrowed to the framework's own
_TraceLevelLoggingHandlerfor the same reason — the previousLoggingHandlermatch would detach an integrator's OTel handler too. Happy to split that into its own commit if you'd rather keep this focused.Tests
Four tests in
tests/test_recording.py. Each was checked by mutation — reverting the corresponding part of the fix makes it fail:keeps_integrator_providers_aliveshutdown()called onceshuts_down_framework_created_provideris_idempotent_across_jobsshutdown()called twice across two teardownsleaves_foreign_log_handlers_attachedLoggingHandleris detached from rootruff check/ruff format --checkclean;scripts/check_types.pyshows only a pre-existingboto3stub error in the AWS plugin.tests/test_recording.pyandtests/test_telemetry_recording_disabled.py: 53 passed.Not addressed here
Two adjacent things I left alone, happy to open issues if useful:
_setup_cloud_traceradds a fresh_MetadataSpanProcessor/BatchSpanProcessorpair to an adopted provider, and the previous job's (now shut down) processors stay attached. Inert, but grows with jobs per process.Proxy/NoOp, so theelsebranch adopts the shut-down provider and spans are dropped. Pre-existing, and orthogonal to ownership — but it means the framework-created path may want to resettracer._tracer_provideron teardown.