PYTHON-6048 Add a sanitizers variant in Evergreen - #22
Draft
blink1073 wants to merge 26 commits into
Draft
Conversation
…n close() TSan caught a data race: Server.close() could return, and the caller could then close monitor sockets, while the monitor's background executor was still mid socket.recv() on the same connection. gc_safe_close() only signals the executor to stop (it must stay non-blocking since it also runs from a GC weakref callback); it never waited for the executor to actually exit before proceeding to reset the pool. Join the executor with the same 1-second bound already used by periodic_executor.py's shutdown path in the three affected close() overrides (MonitorBase, Monitor, _RttMonitor) before touching the pool.
…or join MonitorBase.close()'s join only ever runs for SrvMonitor, which has no pool/socket to race with, so it just added a guaranteed timeout to every mongodb+srv:// client close; drop it and leave the joins in Monitor.close() and _RttMonitor.close(), the classes that actually own the raced socket. Also wake() the executor there before joining so the thread rechecks the stop flag sooner instead of waiting out a full sleep chunk.
…ng GC internals
The TSan task LD_PRELOADed libtsan.so onto a prebuilt, uninstrumented free-threaded interpreter, which reported false races in CPython's own free-threading internals because TSan cannot see synchronization in code compiled without -fsanitize=thread. It now builds CPython 3.14 from source with --with-thread-sanitizer, matching CPython's own CI, so all five suppressions come out. pip keeps build isolation so it resolves hatchling's build dependencies itself; CFLAGS and LDFLAGS are shell env vars, which the isolated build subprocess inherits. The task gets a 7200 second exec timeout because the source build does not fit in the project-wide 3600.
The suppressions file was emptied on the assumption that upstream's Tools/tsan/suppressions_free_threading.txt is empty for an instrumented build. That is true on CPython main but not on the pinned v3.14.0 tag, where it carries 24 entries that upstream's own TSan CI needs. Build a combined file from the cloned tag's copy plus this repo's additions and point TSAN_OPTIONS at that, with handle_segv=0 to match upstream. Also fix the interpreter glob, which missed the real python3.14td name that --disable-gil plus --with-pydebug produces; extend apt-get to CPython's official dependency list; and abort before the 20-30 minute build when the OpenSSL headers pip needs are missing.
Monitor.close() and _RttMonitor.close() no longer block. Joining there burned up to a second while Topology.close() and _update_servers() held the topology lock, stalling every thread in server selection. Instead, extend the existing _monitor_tasks deferral to sync: closed monitors are queued under the lock and joined by cleanup_monitors() once it is released, from Topology.close() and the next select_servers(). Monitor.join() and cleanup_monitors() drop asyncio.gather, which synchro could not translate and which left a broken call in the generated sync monitor.
MongoClient.close() now waits (bounded) for monitor threads to stop before returning, fixing a real race between monitor threads and pool teardown. Document the resulting latency change for users.
blink1073
force-pushed
the
claude/python-6048-jira-ticket-5c9e5c
branch
from
September 2, 2026 01:00
fe8a0d1 to
7ee5382
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PYTHON-6048
Changes in this PR
.evergreen/scripts/run-sanitizer-tests.sh, which builds the driver's C extensions and runstest/test_bson.py,test/test_raw_bson.py,test/test_raw_bson_shared.py, andtest/test_client.pyunder ASan+UBSan (LD_PRELOADed onto a prebuilt interpreter) and under TSan (a fully TSan-instrumented free-threaded CPython built from source, matching CPython's own TSan CI recipe).test-sanitizer-asanandtest-sanitizer-tsanEvergreen tasks and aSanitizersbuild variant, tagged to run on every PR..evergreen/tsan-suppressions.txtfor narrow, justified TSan suppressions, combined at test-run time with the pinned CPython tag's own upstream suppressions file.test/test_raw_bson_shared.pyconcurrency and buffer-lifetime regression tests for zero-copyRawBSONDocument, plus concurrent coverage for the batched wire-message builders inpymongo._cmessage.MongoClient.close()/AsyncMongoClient.close()previously only signaled server monitor background threads/tasks to stop, without waiting for them to actually exit, before tearing down the connection pool.close()now waits (with a bounded timeout, outside any topology lock) for monitors to stop first. This is a narrow, partial implementation of PYTHON-1634; a fuller fix (e.g. joining all executors uniformly, a timeout parameter onclose()) is still tracked there.Monitor.join()usedasyncio.gather(), which doesn't translate to synchronous code, leaving broken code in the generatedpymongo/synchronous/monitor.py.Test Plan
test-sanitizer-asanandtest-sanitizer-tsanEvergreen tasks both pass on a patch build.test/test_client.py,test/test_monitor.py,test/test_bson.py,test/test_raw_bson.py, andtest/test_raw_bson_shared.py(sync and async) all pass locally against a livemongod.just lintandjust typingboth pass.Checklist
Checklist for Author
Checklist for Reviewer