Skip to content

resource_warnings: route unclosed-resource warnings through Salt logger - #70100

Open
dwoz wants to merge 2 commits into
saltstack:3008.xfrom
dwoz:dwoz/fix/log-unclosed-resource-warnings-3008x
Open

resource_warnings: route unclosed-resource warnings through Salt logger#70100
dwoz wants to merge 2 commits into
saltstack:3008.xfrom
dwoz:dwoz/fix/log-unclosed-resource-warnings-3008x

Conversation

@dwoz

@dwoz dwoz commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Python filters ResourceWarning by default, so a bare warnings.warn(..., ResourceWarning) from a __del__ finalizer is silently dropped in production. Callers that missed a close() / destroy() / context-manager contract therefore never see the intended migration signal, and the leaked resource accumulates invisibly.

Concrete incident that motivated this

After commit 0c3f53d9172 removed the __del__-based cleanup from SaltEvent / MasterMinion / RunnerClient / WheelClient in favor of a ResourceWarning-emitting __del__, out-of-tree consumers like SSEAPE that relied on GC-time cleanup via the inline pattern

salt.utils.event.get_master_event(opts, sock_dir).fire_event(...)

began leaking one unix socket per fire-and-forget instance on 3008.2+. In a VMSP master pod the leak accumulated ~350 sockets/min → +180 MB of kernel slab per hour → OOM every ~78 min at the 1 GiB pod cap. The intended ResourceWarning never reached operator logs because ResourceWarning is silenced by default in production Python and Salt didn't route it through logging.

What this PR does

  • New helper salt.utils.resource_warnings.warn_until_close(message, source, category=ResourceWarning, log=None):
    • Emits the ResourceWarning (behavior preserved for dev / test / -W default::ResourceWarning).
    • Also logs the same message at WARNING level via the caller's module logger, so the signal survives Python's default filter and shows up in normal Salt logs.
    • Called only from finalizers — must never raise. Both the warnings.warn and the logger call are wrapped so interpreter-shutdown races don't propagate.
  • Wired into all 8 existing finalizers:
    • salt/utils/event.py (SaltEvent)
    • salt/utils/asynchronous.py (SyncWrapper)
    • salt/transport/tcp.py (Subscriber, TCPPuller, PubServer, _TCPPubServerPublisher)
    • salt/transport/ws.py (PublishClient x2)
  • Removes now-unused import warnings from the 4 files above.
  • Latent bug fix: four of the eight warnings.warn sites (both ws.py sites plus two tcp.py sites) had a missing f prefix, so {self!r} was printed as the literal string instead of interpolating. Fixed as part of the migration.

Test plan

  • New unit tests in tests/pytests/unit/utils/test_resource_warnings.py cover:
    • helper emits ResourceWarning AND a WARNING log record
    • falls back to module logger when no log passed
    • tolerates warnings.warn raising (interpreter-shutdown safety)
    • tolerates a broken logger
    • accepts a custom category (e.g. DeprecationWarning)
  • Standalone verification of behavior (both signals fire, both broken-side cases don't propagate).
  • CI green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:full Run the full test suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant