diff --git a/Doc/library/concurrent.interpreters.rst b/Doc/library/concurrent.interpreters.rst index 058e59b32fd15a..4cae2387e9ed92 100644 --- a/Doc/library/concurrent.interpreters.rst +++ b/Doc/library/concurrent.interpreters.rst @@ -234,6 +234,12 @@ This module defines the following functions: size has been reached, until queue items are consumed. If *maxsize* is less than or equal to zero, the queue size is infinite. +.. function:: is_shareable(obj) + + Return ``True`` if the object can be sent to another interpreter + without using :mod:`pickle`, and ``False`` otherwise. + See :ref:`interp-object-sharing`. + Interpreter objects ^^^^^^^^^^^^^^^^^^^ @@ -275,27 +281,71 @@ Interpreter objects Finalize and destroy the interpreter. - .. method:: prepare_main(ns=None, **kwargs) + .. method:: prepare_main(ns=None, /, **kwargs) + + Bind the given objects into the interpreter's :mod:`!__main__` + module namespace. This is the primary way to pass data to code + running in another interpreter. + + *ns* is an optional :class:`dict` mapping names to values. + Any additional keyword arguments are also bound as names. - Bind objects in the interpreter's :mod:`!__main__` module. + The values must be shareable between interpreters. Some objects + are actually shared, some are copied efficiently, and most are + copied via :mod:`pickle`. See :ref:`interp-object-sharing`. - Some objects are actually shared and some are copied efficiently, - but most are copied via :mod:`pickle`. See :ref:`interp-object-sharing`. + For example:: + + interp = interpreters.create() + interp.prepare_main(name='world') + interp.exec('print(f"Hello, {name}!")') + + This is equivalent to setting variables in the interpreter's + :mod:`!__main__` module before calling :meth:`exec` or :meth:`call`. + The names are available as global variables in the executed code. .. method:: exec(code, /) Run the given source code in the interpreter (in the current thread). + *code* is a :class:`str` of Python source code. It is executed as + though it were the body of a script, using the interpreter's + :mod:`!__main__` module as the globals namespace. + + There is no return value. To get a result back, use :meth:`call` + instead, or communicate through a :class:`Queue`. + + If the code raises an unhandled exception, an :exc:`ExecutionFailed` + exception is raised in the calling interpreter. The actual + exception object is not preserved because objects cannot be shared + between interpreters directly. + + This blocks the current thread until the code finishes. + .. method:: call(callable, /, *args, **kwargs) - Return the result of running the given function in the - interpreter (in the current thread). + Call *callable* in the interpreter (in the current thread) and + return the result. + + Nearly all callables, args, kwargs, and return values are supported. + All "shareable" objects are supported, as are "stateless" functions + (meaning non-closures that do not use any globals). For other + objects, this method falls back to :mod:`pickle`. + + If the callable raises an exception, an :exc:`ExecutionFailed` + exception is raised in the calling interpreter. .. _interp-call-in-thread: .. method:: call_in_thread(callable, /, *args, **kwargs) - Run the given function in the interpreter (in a new thread). + Start a new :class:`~threading.Thread` that calls *callable* in + the interpreter and return the thread object. + + This is a convenience wrapper that combines + :mod:`threading` with :meth:`call`. The thread is started + immediately. Call :meth:`~threading.Thread.join` on the returned + thread to wait for it to finish. Exceptions ^^^^^^^^^^ @@ -338,8 +388,14 @@ Communicating Between Interpreters and :meth:`~queue.Queue.shutdown`. The underlying queue can only be created through :func:`create_queue`. - Some objects are actually shared and some are copied efficiently, - but most are copied via :mod:`pickle`. See :ref:`interp-object-sharing`. + When an object is placed in the queue, it is prepared for use in + another interpreter. Some objects are actually shared and some are + copied efficiently, but most are copied via :mod:`pickle`. + See :ref:`interp-object-sharing`. + + :class:`Queue` objects themselves are shareable between interpreters + (they reference the same underlying queue), making them suitable for + use with :meth:`Interpreter.prepare_main`. .. attribute:: id @@ -347,6 +403,13 @@ Communicating Between Interpreters The queue's ID. + .. attribute:: maxsize + + (read-only) + + The maximum number of items allowed in the queue. A value of zero + means the queue size is infinite. + .. method:: qsize() Return the number of items in the queue. @@ -412,34 +475,60 @@ Creating an interpreter and running code in it:: interp = interpreters.create() - # Run in the current OS thread. + # Run source code directly. + interp.exec('print("Hello from a subinterpreter!")') - interp.exec('print("spam!")') + # Call a function and get the result. + def add(x, y): + return x + y - interp.exec("""if True: - print('spam!') - """) + result = interp.call(add, 3, 4) + print(result) # 7 - from textwrap import dedent - interp.exec(dedent(""" - print('spam!') - """)) + # Run a function in a new thread. + def worker(): + print('Running in a thread!') - def run(arg): - return arg + t = interp.call_in_thread(worker) + t.join() - res = interp.call(run, 'spam!') - print(res) +Passing data with :meth:`~Interpreter.prepare_main`:: - def run(): - print('spam!') + interp = interpreters.create() - interp.call(run) + # Bind variables into the interpreter's __main__ namespace. + interp.prepare_main(greeting='Hello', name='world') + interp.exec('print(f"{greeting}, {name}!")') - # Run in new OS thread. + # Can also use a dict. + config = {'host': 'localhost', 'port': 8080} + interp.prepare_main(config) + interp.exec('print(f"Connecting to {host}:{port}")') - t = interp.call_in_thread(run) - t.join() +Running CPU-bound work in parallel using threads and interpreters:: + + import time + from concurrent import interpreters + + def compute(n): + total = sum(range(n)) + return total + + interp1 = interpreters.create() + interp2 = interpreters.create() + + # Each interpreter runs in its own thread and does not share + # the GIL, enabling true parallel execution. + t1 = interp1.call_in_thread(compute, 50_000_000) + t2 = interp2.call_in_thread(compute, 50_000_000) + t1.join() + t2.join() + +.. tip:: + + For many use cases, :class:`~concurrent.futures.InterpreterPoolExecutor` + provides a higher-level interface that combines threads with + interpreters automatically. Communicating between interpreters with a queue:: diff --git a/Misc/NEWS.d/next/Documentation/2026-03-08-00-00-00.gh-issue-142041.Ab6Luz.rst b/Misc/NEWS.d/next/Documentation/2026-03-08-00-00-00.gh-issue-142041.Ab6Luz.rst new file mode 100644 index 00000000000000..e608a8667a05be --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2026-03-08-00-00-00.gh-issue-142041.Ab6Luz.rst @@ -0,0 +1,8 @@ +Improve :mod:`concurrent.interpreters` documentation with fuller +descriptions of :meth:`~concurrent.interpreters.Interpreter.prepare_main`, +:meth:`~concurrent.interpreters.Interpreter.exec`, +:meth:`~concurrent.interpreters.Interpreter.call` and +:meth:`~concurrent.interpreters.Interpreter.call_in_thread`, document +:func:`~concurrent.interpreters.is_shareable` and +:attr:`Queue.maxsize `, and add more +usage examples.