Repository navigation
Skip the function finalizer once HA has closed the event loop - #897
Conversation
The weakref finalizer on a function's EvalFuncVar also ran from atexit, after HA had closed the loop, so a manager still RUNNING called hass.async_create_task() on a closed loop and logged a traceback at shutdown. Don't run it at exit, and return early when the loop is already closed. Fixes #880.
There was a problem hiding this comment.
Thanks, this looks good. Not running the finalizer at exit and skipping it once the loop is closed address #880.
Optional: make the finalizer thread-safe
There's a related problem that was already there before this PR. Many pyscript objects are referenced in cycles, so they're freed by the cyclic garbage collector, which runs in whichever thread triggers it, including HA's executor threads. The finalizer then calls hass.async_create_task() off the event loop. I reproduced this by deleting a running manager's function in an executor job and calling gc.collect() there. HA raised its thread-safety error (Detected that custom integration 'pyscript' calls hass.async_create_task from a thread other than the event loop ...), and the manager was never stopped. It also leaves a small gap where the loop can close between the is_closed() check and async_create_task().
Handing off with call_soon_threadsafe fixes both, because it's safe from any thread and raises RuntimeError if the loop is already closed:
def stop_deleted_func():
if self.status is DecoratorManagerStatus.RUNNING:
self.hass.async_create_task(self.safe_await(self.stop()))
def on_func_var_deleted():
# garbage collection can run this in any thread, so hand off to the loop
try:
self.hass.loop.call_soon_threadsafe(stop_deleted_func)
except RuntimeError:
# once HA has closed the loop at shutdown there is nothing left to stop
passYour atexit = False stays as is. I tried this locally: your three tests pass unchanged, and this test (which fails on the current branch) passes:
@pytest.mark.asyncio
async def test_function_decorator_manager_finalizer_in_worker_thread(hass, caplog):
"""Deleting the function in a worker thread should stop the manager via the loop."""
DecoratorManager.hass = hass
holder = [DummyEvalFuncVar()]
manager = FunctionDecoratorManager(DummyAstCtx(), holder[0])
manager.update_status(DecoratorManagerStatus.RUNNING)
def drop():
holder.clear()
gc.collect()
await hass.async_add_executor_job(drop)
await hass.async_block_till_done()
assert manager.status is DecoratorManagerStatus.STOPPED
assert "thread other than the event loop" not in caplog.textIt's up to you whether to add this here. If you'd rather keep this PR focused, just say so and I'll merge it as is and add the thread-safety change in a follow-up commit.
| def on_func_var_deleted(): | ||
| if self.status is DecoratorManagerStatus.RUNNING: | ||
| # once HA has closed the loop at shutdown there is nothing left to stop | ||
| if self.status is DecoratorManagerStatus.RUNNING and not self.hass.loop.is_closed(): |
There was a problem hiding this comment.
Optional (see the review summary): this can run in any thread that triggers garbage collection, and the loop can close between this check and async_create_task(). Handing off with hass.loop.call_soon_threadsafe(...) and catching RuntimeError covers both.
The cyclic garbage collector can run the weakref finalizer in any thread, including HA's executor threads, where calling hass.async_create_task() trips HA's thread-safety check and the manager is never stopped. Schedule the stop through loop.call_soon_threadsafe() instead, which also raises RuntimeError once the loop is closed, so the separate is_closed() check and its race go away. As suggested in review.
|
Thanks for digging into that! I've added your change and the worker-thread test in a second commit. Full suite passes (199), and the new test fails without the change. |
|
Looks good - thanks for the PR! |
FunctionDecoratorManagerregisters aweakref.finalizeon the function so a running manager gets stopped when the function goes away.weakref.finalizealso runs from atexit by default, and at that point HA has already closed the loop, so a manager that was still RUNNING callshass.async_create_task()on a closed loop and leaves the "Event loop is closed" traceback from #880 (plus the "Task was destroyed but it is pending" line for the coroutine that never ran).The finalizer now has
atexit = False, since there is nothing left to stop after shutdown. It also hands the stop off withhass.loop.call_soon_threadsafe(), because the cyclic GC can run it in any thread (including HA's executor threads), and that call raisesRuntimeErroronce the loop is closed. That covers the function being collected after the loop closed but before the interpreter exits. Normal deletion while HA is running still stops the manager.Added tests in
test_decorator_manager.py: deleting the function of a running manager stops it, a running manager whose loop is closed doesn't callasync_create_task, the finalizer is not registered for atexit, and deleting the function in a worker thread stops the manager through the loop without HA's thread-safety warning. The closed-loop and worker-thread tests fail without the change. Full test suite passes. I haven't reproduced the shutdown traceback on a real HA install, it's intermittent for the reporter.Fixes #880