Repository navigation
Fail any test that leaks a PeriodicExportingMetricReader ticker thread, and fix the four that do - #5641
Fail any test that leaks a PeriodicExportingMetricReader ticker thread, and fix the four that do#5641ocelotl wants to merge 5 commits into
Conversation
Pull request dashboard statusWaiting on the author · refreshed 2026-10-20 23:06 UTC Respond to 2 review items (e.g. link a commit, explain why not, ask a follow-up): Status above doesn't look right?
|
| mock_exporter._preferred_temporality = None | ||
| mock_exporter._preferred_aggregation = None | ||
| metric_reader_0 = PeriodicExportingMetricReader(mock_exporter) | ||
| self.addCleanup(metric_reader_0.shutdown) |
There was a problem hiding this comment.
These cleanups shut down the readers directly, but the MeterProvider instances still register atexit handlers that call them again. Could the provider constructions below use shutdown_on_exit=False to avoid leaving those callbacks behind?
| @@ -0,0 +1 @@ | |||
| `opentelemetry-sdk`: fail any test that leaves a `PeriodicExportingMetricReader` ticker thread running, and shut down the four readers that were leaking one into the rest of the test session | |||
There was a problem hiding this comment.
| `opentelemetry-sdk`: fail any test that leaves a `PeriodicExportingMetricReader` ticker thread running, and shut down the four readers that were leaking one into the rest of the test session | |
| `opentelemetry-sdk`: detect and clean up leaked periodic metric reader threads in tests |
|
Hi @ocelotl — just a friendly reminder that this pull request is waiting on you. The dashboard status comment has the open items and is kept current.
|
|
This PR has been automatically marked as stale because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 days of this comment. |
The test registers a PeriodicExportingMetricReader on a MeterProvider and never shuts either down, so the reader's daemon ticker thread stays alive for the rest of the session and keeps calling collect() during unrelated tests. The provider holds an observable counter, so every one of those later ticks runs the callback and feeds a measurement back into the storage. Register the shutdown with addCleanup so it runs even when an assertion fails. Refs open-telemetry#5640
The test builds two PeriodicExportingMetricReader instances to check that a reader cannot be registered on two MeterProviders. Neither reader is ever shut down, so both ticker threads outlive the test. Register both shutdowns with addCleanup. The readers, not the providers, because the providers here are deliberately transient and one of them is expected to fail construction. Refs open-telemetry#5640
test_exporter_temporality_preference and test_exporter_aggregation_preference each build a PeriodicExportingMetricReader purely to inspect its preference mapping, and never shut it down. The reader is not registered on a MeterProvider, so its ticks return early and cannot corrupt another test's state, but each one still leaves a thread running for the rest of the session and logs a warning every export interval into whatever test is running. Refs open-telemetry#5640
PeriodicExportingMetricReader starts a daemon ticker thread in its constructor, and nothing reaps a daemon thread between tests. A reader that is never shut down leaves that thread waking on its own clock for the rest of the session, calling collect() inside unrelated tests. With the default 60 second interval the first tick lands long after the guilty test has passed, so the failure surfaces somewhere else and names the victim rather than the cause. Add an autouse fixture that snapshots the live ticker threads before each test and fails if a new one is still alive afterwards. Diffing against the snapshot attributes the leak to the test that caused it and stops one leak from cascading into every test that follows. Poll for a short grace period first, because shutdown() joins the ticker with a timeout and a thread on its way out can still be alive when the test body returns. Only a test that actually leaks pays that cost. The three commits before this one fix the leaks that already exist, so this lands on a clean suite. Closes open-telemetry#5640
7316375 to
1b17f13
Compare
Description
Fixes #5640
PeriodicExportingMetricReaderstarts a daemon ticker thread in its constructor, and nothing reaps a daemon thread between tests. A reader that is never shut down leaves that thread waking on its own clock for the rest of the session, callingcollect()inside unrelated tests. With the default 60 second interval its first tick lands long after the guilty test has passed, so the guilty test is green and the damage shows up somewhere else, naming the victim rather than the cause.There was no check for this. This PR adds one, and fixes the four leaks that exist today so it can land on a clean suite.
Commits
Each commit touches exactly one file.
test_exporter_not_called_concurrentlyregisters a reader on aMeterProviderand never shuts either down. This is the most consequential of the four: the provider holds an observable counter, so every later tick runs the callback and feeds a measurement back into the storage.test_register_metric_readersbuilds two readers to check that one cannot be registered on two providers, and shuts down neither. Two leaked threads.self._collect is Noneguard and cannot corrupt another test's state, but each still burns a thread and logs a warning every interval into whatever test is running.opentelemetry-sdk/tests/conftest.py.All four fixes use
addCleanupso the shutdown happens even when the test body fails.How the detector works
It snapshots the live threads named
OtelPeriodicExportingMetricReaderbefore the test and fails if a new one is still alive afterwards.Diffing against a before-snapshot is the important part. It attributes a leak to the test that caused it, and it stops one leak from cascading into a failure in every test that runs after it.
There is a short grace period before it gives up, because
shutdown()joins the ticker with a timeout and a thread on its way out can still be alive when the test body returns. Only a test that actually leaks pays that cost.Testing
pytest opentelemetry-sdk/testsbefore the fixes, with the detector in place: 4 errors, naming exactly the four tests above.pytest opentelemetry-sdk/testsafter: 888 passed, 47 subtests passed, 0 errors.ruff format --checkandruff checkclean on all four changed files with the pinnedruff==0.16.1.I also checked that the detector fires and does not cascade, using a scratch module with two tests: one that builds a reader and drops it, one that builds a reader and shuts it down. Result was
2 passed, 1 error, the error naming only the leaking test. The well-behaved test ran afterwards and passed, confirming the before-snapshot prevents the cascade.Scope and overlap
Test-only. No production code touched.
Commit 3 overlaps with #5622, which fixes the same two preference tests with
try/finallyrather thanaddCleanup. Happy to drop that commit if #5622 lands first, though the detector needs those two fixed one way or the other to stay green.Relationship to #5157
I want to be careful not to overclaim. I have not proven that any of these four leaks produced the traceback in #5157; the CI logs from that run have expired, and the surviving excerpt does not pin it down. What this gives is a mechanical guard, so that if it recurs the leaking test fails by name instead of the symptom surfacing three files later.
The other half of #5157, the tests asserting on process-global mock state, is #5638 and #5639.