test_external_inspection.test_only_active_thread is flaky

[colesbury]

Bug report

Bug description:

test test_external_inspection failed -- Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython/Lib/test/test_external_inspection.py", line 991, in test_only_active_thread
    self.assertEqual(len(gil_traces), 1, "Should have exactly one GIL holder")
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 0 != 1 : Should have exactly one GIL holder

See the failures in https://github.com/python/cpython/actions/runs/16006058028/job/45152923015 on main

cc @pablogsal

See also:

• gh-91048: Fix external inspection multi-threaded performance #136005

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

• gh-136186: Fix race condition in test_external_inspection.test_only_active_thread #136347
• [3.14] gh-136186: Fix race condition in test_external_inspection.test_only_active_thread (GH-136347) #136416
• gh-136186: Fix flaky tests in test_external_inspection #143110
• gh-136186: Fix more flaky tests in test_external_inspection #143235

[pablogsal]

Will take a look 👀

[StanFromIreland]

Also fails on buildbots, e.g. this run

[vstinner]

I can reproduce the issue on Linux with the command:

$ ./python -m test test_external_inspection -v -m test_only_active_thread -F -j40
(...)
AssertionError: 0 != 1 : Should have exactly one GIL holder

[sergey-miryanov]

I can reproduce the issue on Linux with the command:

$ ./python -m test test_external_inspection -v -m test_only_active_thread -F -j40
(...)
AssertionError: 0 != 1 : Should have exactly one GIL holder

With this command I got following error on Windows (JFYI):

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\Sources\\_pythonish\\cpython\\main\\build\\test_python_50684\xe6'

0:00:11 load avg: 59.23 [ 11/1] test_external_inspection worker non-zero exit code (Exit code 1)
test_only_active_thread (test.test_external_inspection.TestGetStackTrace.test_only_active_thread) ... ok

----------------------------------------------------------------------
Ran 1 test in 3.596s

OK
Traceback (most recent call last):
  File "D:\Sources\_pythonish\cpython\main\Lib\test\support\os_helper.py", line 540, in temp_dir
    yield path
  File "D:\Sources\_pythonish\cpython\main\Lib\test\support\os_helper.py", line 599, in temp_cwd
    yield cwd_dir
  File "D:\Sources\_pythonish\cpython\main\Lib\test\libregrtest\worker.py", line 127, in main
    worker_process(worker_json)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\test\libregrtest\worker.py", line 113, in worker_process
    sys.exit(0)
    ~~~~~~~~^^^
SystemExit: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Sources\_pythonish\cpython\main\Lib\test\support\__init__.py", line 208, in _force_run
    return func(*args)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\Sources\\_pythonish\\cpython\\main\\build\\test_python_50684\xe6'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Sources\_pythonish\cpython\main\Lib\runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
                     "__main__", mod_spec)
  File "D:\Sources\_pythonish\cpython\main\Lib\runpy.py", line 88, in _run_code
    exec(code, run_globals)
    ~~~~^^^^^^^^^^^^^^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\test\libregrtest\worker.py", line 131, in <module>
    main()
    ~~~~^^
  File "D:\Sources\_pythonish\cpython\main\Lib\test\libregrtest\worker.py", line 126, in main
    with os_helper.temp_cwd(work_dir, quiet=True):
         ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\test\support\os_helper.py", line 597, in temp_cwd
    with temp_dir(path=name, quiet=quiet) as temp_path:
         ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\test\support\os_helper.py", line 545, in temp_dir
    rmtree(path)
    ~~~~~~^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\test\support\os_helper.py", line 497, in rmtree
    _rmtree(path)
    ~~~~~~~^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\test\support\os_helper.py", line 441, in _rmtree
    _waitfor(lambda p: _force_run(p, os.rmdir, p), path)
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\test\support\os_helper.py", line 381, in _waitfor
    func(pathname)
    ~~~~^^^^^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\test\support\os_helper.py", line 441, in <lambda>
    _waitfor(lambda p: _force_run(p, os.rmdir, p), path)
                       ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\test\support\__init__.py", line 219, in _force_run
    return func(*args)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\Sources\\_pythonish\\cpython\\main\\build\\test_python_50684\xe6'
Kill <WorkerThread #1 running test=test_external_inspection pid=67748 time=11.2 sec> process
Kill <WorkerThread #2 running test=test_external_inspection pid=75760 time=11.2 sec> process
Kill <WorkerThread #3 running test=test_external_inspection pid=37312 time=11.2 sec> process
Kill <WorkerThread #4 running test=test_external_inspection pid=54008 time=136 ms> process
Kill <WorkerThread #5 running test=test_external_inspection pid=71532 time=11.2 sec> process
Kill <WorkerThread #6 running test=test_external_inspection pid=75024 time=908 ms> process
Kill <WorkerThread #7 running test=test_external_inspection pid=76908 time=11.2 sec> process
Kill <WorkerThread #8 running test=test_external_inspection pid=67792 time=11.2 sec> process
Kill <WorkerThread #9 running test=test_external_inspection pid=77664 time=111 ms> process
Kill <WorkerThread #11 running test=test_external_inspection pid=77316 time=11.2 sec> process
Kill <WorkerThread #13 running test=test_external_inspection pid=77100 time=11.2 sec> process
Kill <WorkerThread #14 running test=test_external_inspection pid=67208 time=11.1 sec> process
Kill <WorkerThread #15 running test=test_external_inspection pid=70904 time=11.1 sec> process
Kill <WorkerThread #16 running test=test_external_inspection pid=76680 time=2.4 sec> process
Kill <WorkerThread #17 running test=test_external_inspection pid=69020 time=11.1 sec> process
Kill <WorkerThread #18 running test=test_external_inspection pid=68684 time=1.8 sec> process
Kill <WorkerThread #20 running test=test_external_inspection pid=71156 time=973 ms> process
Kill <WorkerThread #21 running test=test_external_inspection pid=77156 time=11.1 sec> process
Kill <WorkerThread #22 running test=test_external_inspection pid=62752 time=11.1 sec> process
Kill <WorkerThread #23 running test=test_external_inspection pid=74376 time=11.1 sec> process
Kill <WorkerThread #24 running test=test_external_inspection pid=59932 time=1.8 sec> process
Kill <WorkerThread #25 running test=test_external_inspection pid=71704 time=11.1 sec> process
Kill <WorkerThread #26 running test=test_external_inspection pid=76612 time=11.1 sec> process
Kill <WorkerThread #27 running test=test_external_inspection pid=76964 time=2.3 sec> process
Kill <WorkerThread #28 running test=test_external_inspection pid=76940 time=11.1 sec> process
Kill <WorkerThread #29 running test=test_external_inspection pid=77228 time=1.9 sec> process
Kill <WorkerThread #30 running test=test_external_inspection pid=56868 time=11.1 sec> process
Kill <WorkerThread #31 running test=test_external_inspection pid=76344 time=11.1 sec> process
Kill <WorkerThread #32 running test=test_external_inspection pid=77052 time=11.1 sec> process
Kill <WorkerThread #33 running test=test_external_inspection pid=61188 time=11.1 sec> process
Kill <WorkerThread #34 running test=test_external_inspection pid=60384 time=11.1 sec> process
Kill <WorkerThread #35 running test=test_external_inspection pid=74884 time=107 ms> process
Kill <WorkerThread #36 running test=test_external_inspection pid=36240 time=11.0 sec> process
Kill <WorkerThread #37 running test=test_external_inspection pid=77264 time=11.0 sec> process
Kill <WorkerThread #38 running test=test_external_inspection pid=77176 time=11.0 sec> process
Kill <WorkerThread #39 running test=test_external_inspection pid=68476 time=11.1 sec> process
Kill <WorkerThread #40 running test=test_external_inspection pid=70452 time=11.0 sec> process

== Tests result: FAILURE ==

1 test failed:
    test_external_inspection

[pablogsal]

I can reproduce the issue on Linux with the command:

$ ./python -m test test_external_inspection -v -m test_only_active_thread -F -j40
(...)
AssertionError: 0 != 1 : Should have exactly one GIL holder

With this command I got following error on Windows (JFYI):

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\Sources\\_pythonish\\cpython\\main\\build\\test_python_50684\xe6'

0:00:11 load avg: 59.23 [ 11/1] test_external_inspection worker non-zero exit code (Exit code 1)
test_only_active_thread (test.test_external_inspection.TestGetStackTrace.test_only_active_thread) ... ok

----------------------------------------------------------------------
Ran 1 test in 3.596s

OK
Traceback (most recent call last):
  File "D:\Sources\_pythonish\cpython\main\Lib\test\support\os_helper.py", line 540, in temp_dir
    yield path
  File "D:\Sources\_pythonish\cpython\main\Lib\test\support\os_helper.py", line 599, in temp_cwd
    yield cwd_dir
  File "D:\Sources\_pythonish\cpython\main\Lib\test\libregrtest\worker.py", line 127, in main
    worker_process(worker_json)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\test\libregrtest\worker.py", line 113, in worker_process
    sys.exit(0)
    ~~~~~~~~^^^
SystemExit: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Sources\_pythonish\cpython\main\Lib\test\support\__init__.py", line 208, in _force_run
    return func(*args)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\Sources\\_pythonish\\cpython\\main\\build\\test_python_50684\xe6'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Sources\_pythonish\cpython\main\Lib\runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
                     "__main__", mod_spec)
  File "D:\Sources\_pythonish\cpython\main\Lib\runpy.py", line 88, in _run_code
    exec(code, run_globals)
    ~~~~^^^^^^^^^^^^^^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\test\libregrtest\worker.py", line 131, in <module>
    main()
    ~~~~^^
  File "D:\Sources\_pythonish\cpython\main\Lib\test\libregrtest\worker.py", line 126, in main
    with os_helper.temp_cwd(work_dir, quiet=True):
         ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\test\support\os_helper.py", line 597, in temp_cwd
    with temp_dir(path=name, quiet=quiet) as temp_path:
         ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\test\support\os_helper.py", line 545, in temp_dir
    rmtree(path)
    ~~~~~~^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\test\support\os_helper.py", line 497, in rmtree
    _rmtree(path)
    ~~~~~~~^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\test\support\os_helper.py", line 441, in _rmtree
    _waitfor(lambda p: _force_run(p, os.rmdir, p), path)
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\test\support\os_helper.py", line 381, in _waitfor
    func(pathname)
    ~~~~^^^^^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\test\support\os_helper.py", line 441, in <lambda>
    _waitfor(lambda p: _force_run(p, os.rmdir, p), path)
                       ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "D:\Sources\_pythonish\cpython\main\Lib\test\support\__init__.py", line 219, in _force_run
    return func(*args)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\Sources\\_pythonish\\cpython\\main\\build\\test_python_50684\xe6'
Kill <WorkerThread #1 running test=test_external_inspection pid=67748 time=11.2 sec> process
Kill <WorkerThread #2 running test=test_external_inspection pid=75760 time=11.2 sec> process
Kill <WorkerThread #3 running test=test_external_inspection pid=37312 time=11.2 sec> process
Kill <WorkerThread #4 running test=test_external_inspection pid=54008 time=136 ms> process
Kill <WorkerThread #5 running test=test_external_inspection pid=71532 time=11.2 sec> process
Kill <WorkerThread #6 running test=test_external_inspection pid=75024 time=908 ms> process
Kill <WorkerThread #7 running test=test_external_inspection pid=76908 time=11.2 sec> process
Kill <WorkerThread #8 running test=test_external_inspection pid=67792 time=11.2 sec> process
Kill <WorkerThread #9 running test=test_external_inspection pid=77664 time=111 ms> process
Kill <WorkerThread #11 running test=test_external_inspection pid=77316 time=11.2 sec> process
Kill <WorkerThread #13 running test=test_external_inspection pid=77100 time=11.2 sec> process
Kill <WorkerThread #14 running test=test_external_inspection pid=67208 time=11.1 sec> process
Kill <WorkerThread #15 running test=test_external_inspection pid=70904 time=11.1 sec> process
Kill <WorkerThread #16 running test=test_external_inspection pid=76680 time=2.4 sec> process
Kill <WorkerThread #17 running test=test_external_inspection pid=69020 time=11.1 sec> process
Kill <WorkerThread #18 running test=test_external_inspection pid=68684 time=1.8 sec> process
Kill <WorkerThread #20 running test=test_external_inspection pid=71156 time=973 ms> process
Kill <WorkerThread #21 running test=test_external_inspection pid=77156 time=11.1 sec> process
Kill <WorkerThread #22 running test=test_external_inspection pid=62752 time=11.1 sec> process
Kill <WorkerThread #23 running test=test_external_inspection pid=74376 time=11.1 sec> process
Kill <WorkerThread #24 running test=test_external_inspection pid=59932 time=1.8 sec> process
Kill <WorkerThread #25 running test=test_external_inspection pid=71704 time=11.1 sec> process
Kill <WorkerThread #26 running test=test_external_inspection pid=76612 time=11.1 sec> process
Kill <WorkerThread #27 running test=test_external_inspection pid=76964 time=2.3 sec> process
Kill <WorkerThread #28 running test=test_external_inspection pid=76940 time=11.1 sec> process
Kill <WorkerThread #29 running test=test_external_inspection pid=77228 time=1.9 sec> process
Kill <WorkerThread #30 running test=test_external_inspection pid=56868 time=11.1 sec> process
Kill <WorkerThread #31 running test=test_external_inspection pid=76344 time=11.1 sec> process
Kill <WorkerThread #32 running test=test_external_inspection pid=77052 time=11.1 sec> process
Kill <WorkerThread #33 running test=test_external_inspection pid=61188 time=11.1 sec> process
Kill <WorkerThread #34 running test=test_external_inspection pid=60384 time=11.1 sec> process
Kill <WorkerThread #35 running test=test_external_inspection pid=74884 time=107 ms> process
Kill <WorkerThread #36 running test=test_external_inspection pid=36240 time=11.0 sec> process
Kill <WorkerThread #37 running test=test_external_inspection pid=77264 time=11.0 sec> process
Kill <WorkerThread #38 running test=test_external_inspection pid=77176 time=11.0 sec> process
Kill <WorkerThread #39 running test=test_external_inspection pid=68476 time=11.1 sec> process
Kill <WorkerThread #40 running test=test_external_inspection pid=70452 time=11.0 sec> process

== Tests result: FAILURE ==

1 test failed:
    test_external_inspection

That's unfortunately another problem that's unique for windows. If you check the bug tracker you will see is a race between processes and the temporary directory