Skip to content

Segfault in test_sampling_profiler #142973

New issue
New issue
Closed as duplicate
Closed as duplicate
Segfault in test_sampling_profiler#142973
Labels
testsTests in the Lib/test dirtopic-profilingtype-crashA hard crash of the interpreter, possibly with a core dump

Description

@encukou
encukou
opened on Dec 19, 2025

Crash report

What happened?

A recent Buildbot run (AMD64 Debian root) saw a segfault in test_sampling_profiler: https://buildbot.python.org/#/builders/345/builds/12988

test_sampling_all_threads (test.test_profiling.test_sampling_profiler.test_integration.TestSampleProfilerIntegration.test_sampling_all_threads) ... Fatal Python error: Segmentation fault

Current thread 0x00007f85531fa740 [python] (most recent call first):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/profiling/sampling/sample.py", line 75 in sample
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/profiling/sampling/sample.py", line 350 in sample
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_profiling/test_sampling_profiler/test_integration.py", line 521 in test_sampling_all_threads
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/case.py", line 613 in _callTestMethod
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/case.py", line 667 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/case.py", line 723 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/runner.py", line 257 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 84 in _run_suite
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 42 in run_unittest
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 162 in test_func
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 118 in regrtest_runner
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 165 in _load_run_test
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 210 in _runtest_env_changed_exc
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 319 in _runtest
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/single.py", line 348 in run_single_test
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/worker.py", line 99 in worker_process
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/worker.py", line 134 in main
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/worker.py", line 138 in <module>
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/runpy.py", line 87 in _run_code
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/runpy.py", line 196 in _run_module_as_main

Current thread's C stack trace (most recent call first):
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at _Py_DumpStack+0x33 [0x563947ed22d4]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x3285b2 [0x563947eea5b2]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x328caa [0x563947eeacaa]
  Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x3bfd0 [0x7f8553238fd0]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at _PyEval_EvalFrameDefault+0x5795 [0x563947e2bc9a]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x280c92 [0x563947e42c92]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x280e44 [0x563947e42e44]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at _PyFunction_Vectorcall+0x54 [0x563947cee2da]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x12f1c1 [0x563947cf11c1]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x12f326 [0x563947cf1326]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x12da86 [0x563947cefa86]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x12dd76 [0x563947cefd76]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at PyObject_Call+0x20 [0x563947cefdae]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at _PyEval_EvalFrameDefault+0x60a2 [0x563947e2c5a7]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x280c92 [0x563947e42c92]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x280e44 [0x563947e42e44]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at _PyFunction_Vectorcall+0x54 [0x563947cee2da]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x12d7d3 [0x563947cef7d3]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x12d99d [0x563947cef99d]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x1c26af [0x563947d846af]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x1c278d [0x563947d8478d]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at _PyObject_MakeTpCall+0xce [0x563947cee47c]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x12c6af [0x563947cee6af]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at PyObject_Vectorcall+0x23 [0x563947cee6fd]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at _Py_VectorCall_StackRefSteal+0x86 [0x563947e237d2]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at _PyEval_EvalFrameDefault+0x90b2 [0x563947e2f5b7]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x280c92 [0x563947e42c92]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x280e44 [0x563947e42e44]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at _PyFunction_Vectorcall+0x54 [0x563947cee2da]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x12f1c1 [0x563947cf11c1]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x12f326 [0x563947cf1326]
  Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x12da86 [0x563947cefa86]
  <truncated rest of calls>

A similar crash happened earlier in the run in test_sampling_with_collapsed_export.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

No response

Activity

  1. encukou
    added
    type-crashA hard crash of the interpreter, possibly with a core dump
    topic-profiling
    on Dec 19, 2025
  2. encukou
    changed the title [-]`test_sampling_profiler`[/-] [+]Segfault in `test_sampling_profiler`[/+] on Dec 19, 2025

  3. fatelei commented on Dec 22, 2025

    @fatelei
    fatelei
    on Dec 22, 2025
    Contributor

    use main branch, test in local the test pass

    ./python -m test -v test_profiling | grep test_sampling_all_threads
test_sampling_all_threads (test.test_profiling.test_sampling_profiler.TestSampleProfilerIntegration.test_sampling_all_threads) ... ok

  4. pablogsal commented on Dec 26, 2025

    @pablogsal
    pablogsal
    on Dec 26, 2025
    Member

    I cannot reproduce this. Unfortunately the c stack output doesn't point at anything useful either :(

  5. pablogsal commented on Dec 26, 2025

    @pablogsal
    pablogsal
    on Dec 26, 2025
    Member

    Also I am relatively sure this is not the profiler because the buildbot is testing commit 220f0b1 and crashing here:

    Current thread 0x00007f85531fa740 [python] (most recent call first):  
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/profiling/sampling/sample.py", line 75 in sample
 File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/profiling/sampling/sample.py", line 350 in sample
 File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_profiling/test_sampling_profiler/test_integration.py", line 521 in test_sampling_all_threads
 File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/case.py", line 613 in _callTestMethod

    but in that commit line 75 corresponds to:

    cpython/Lib/profiling/sampling/sample.py

    Line 75 in 220f0b1

    if hasattr(collector, 'running') and not collector.running:

    which is not even the C code of the profiler it's some random Python code. And indeed, the C stack shows that this crashes in the eval loop:

    Current thread's C stack trace (most recent call first):  
Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at _Py_DumpStack+0x33 [0x563947ed22d4] 
Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x3285b2 [0x563947eea5b2] 
Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x328caa [0x563947eeacaa]  
Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x3bfd0 [0x7f8553238fd0]  
Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at _PyEval_EvalFrameDefault+0x5795 [0x563947e2bc9a]  
Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x280c92 [0x563947e42c92]  
Binary file "/root/buildarea/3.x.angelico-debian-amd64/build/python", at +0x280e44 [0x563947e42e44]

    So whatever it is I am quite sure is not the profiler

  6. serhiy-storchaka
    added
    testsTests in the Lib/test dir
    on Jul 4, 2026

  7. hugovk commented on Oct 29, 2026

    @hugovk
    hugovk
    on Oct 29, 2026
    Member

    Probably a hardware fault in the buildbot? See #159764.

  8. Rosuav commented on Oct 29, 2026

    @Rosuav
    Rosuav
    on Oct 29, 2026
    Contributor

    Yeah, this looks like the same fault as #159764. I'm currently working on a warranty replacement for the CPU, but in the meantime, I've pinned the buildbot VM to a ocuple of eco cores that seem to work correctly; builds are a bit slower (I think it's 25%-50% by the look of things) but seem to be working reliably.

  9. hugovk commented on Oct 29, 2026

    @hugovk
    hugovk
    on Oct 29, 2026
    Member

    Thanks, let's close this in favour of that issue with all the investigation.

  10. hugovk
    closed this as a duplicateof Python 3.14 crash randomly on AMD64 Debian root 3.14 buildbot: CPU bug #159764 on Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testsTests in the Lib/test dirtopic-profilingtype-crashA hard crash of the interpreter, possibly with a core dump

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions