Skip to content

Executor.map() exceptions lose the worker traceback and form a cycle #158497

Open

Description

@fedonman

Bug description:

Since gh-108518, the exception that Executor.map() re-raises for a failed call differs from 3.15 in two ways:

  • ProcessPoolExecutor.map() no longer sets the worker-side traceback as its __cause__, so the traceback only shows raise exc in _MapResultIterator.__next__.
  • For every executor, the exception is kept alive by a reference cycle through the __next__ frame, together with the frames and locals of the failed call, until the cyclic GC runs.
import gc, weakref
from concurrent.futures import ProcessPoolExecutor

class E(Exception):
    pass

def f(x):
    raise E(x)

if __name__ == '__main__':
    gc.disable()
    with ProcessPoolExecutor() as ex:
        try:
            next(ex.map(f, [1]))
        except E as e:
            print('cause:', type(e.__cause__).__name__)
            ref = weakref.ref(e)
    print('freed:', ref() is None)

On main:

cause: NoneType
freed: False

Expected, as on 3.15:

cause: _RemoteTraceback
freed: True

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

Metadata

Metadata

Assignees

No one assigned

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytopic-multiprocessingtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions