fix: drain poll_task after cancel to prevent RuntimeError on closed loop in ActivityWorker/NexusWorker

[kenanzamp]

fix: drain poll_task after cancel to prevent RuntimeError("Event loop is closed") in ActivityWorker and NexusWorker

Summary

_ActivityWorker.run() and _NexusWorker.run() both follow this shutdown pattern when a worker exception fires:

poll_task.cancel()
await exception_task   # re-raises the worker error

poll_task.cancel() schedules a CancelledError delivery but returns immediately — it does not wait for the loop to process it. Meanwhile, the Rust bridge (pyo3-async-runtimes) may still have a result on its way via call_soon_threadsafe, which places a callback in the event loop's _ready queue.

At this point every Python-visible asyncio.Task is done. asyncio.run()'s _cancel_all_tasks sweep finds nothing to cancel and calls loop.close() immediately. If loop.close() races with the pending call_soon_threadsafe callback, Python raises:

RuntimeError: Event loop is closed

This race existed before v1.13.0 but the window was tiny. The v1.13.0 Worker plugin-chain refactor (introduced in #1337 / 5e93e63) wrapped _ActivityWorker.run() in additional async layers, consistently widening the window and making the race reliably reproducible.

Root cause

The race is entirely on the Python side — it lives in the interaction between asyncio.Task.cancel(), asyncio.run() teardown, and call_soon_threadsafe. No changes to sdk-core (Rust) are required or proposed.

Fix

Insert await asyncio.wait([poll_task]) immediately after poll_task.cancel():

poll_task.cancel()
await asyncio.wait([poll_task])   # drain: lets any pending Rust callback fire while loop is still open
await exception_task

asyncio.wait() yields to the event loop for at least one iteration, flushing any queued call_soon_threadsafe callbacks before loop.close() is called. It does not raise on a cancelled task. The same fix is applied symmetrically to _NexusWorker.run().

Testing

tests/worker/test_shutdown_race.py (new file) reproduces the race deterministically by patching call_soon_threadsafe to defer callbacks until after poll_task.cancel() returns. Without the fix, 10/10 iterations raise RuntimeError("Event loop is closed"). With the fix, all iterations complete cleanly.

Workaround context

This fix resolves a race first observed in production at Zamp (see workaround at https://github.com/Zampfi/pantheon/pull/5329). The workaround wrapped the worker in a retry loop; this fix removes the need for that workaround by addressing the root cause in the SDK itself.

Checklist

• Reproducer confirmed (see tests/worker/test_shutdown_race.py)
• Fix applied to _activity.py and _nexus.py
• ruff check --select I — clean
• ruff format --check — clean
• mypy — pre-existing error in workflow_sandbox/_importer.py (unrelated, present on main)
• No changes to sdk-core (Rust)
• CLA: @kenanzamp (human) will sign the Temporal CLA

Fixes the RuntimeError: Event loop is closed shutdown race made consistently reproducible in v1.13.0.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 2 committers have signed the CLA.

❌ kenanzamp
❌ pace-bot
_{You have signed the CLA already but the status is still pending? Let us recheck it.}