Skip to content

SocketModeClient (aiohttp backend): connect() retries forever against a closed ClientSession — orphaned retry loop survives close() #1913

Description

@eBeat-developers

Environment

  • slack_sdk 3.40.1, slack_bolt 1.27.0 (AsyncSocketModeHandler), aiohttp 3.14.1
  • Python 3.11, macOS, long-running gateway process under launchd

Summary

slack_sdk.socket_mode.aiohttp.SocketModeClient creates its aiohttp.ClientSession once in __init__ (line 130) and never recreates it. The connect() retry loop (lines 347–409) catches every exception, sleeps ping_interval (default 10s), and retries — without ever checking self.closed or whether the session object is still usable. Once that loop runs against a closed session, ws_connect() raises RuntimeError: Session is closed immediately on every attempt, so the loop can never succeed and never exits: a permanent retry loop for the lifetime of the process.

ERROR slack_bolt.AsyncApp: Failed to connect (error: Session is closed); Retrying...
Traceback (most recent call last):
  File ".../slack_sdk/socket_mode/aiohttp/__init__.py", line 377, in connect
    self.current_session = await self.aiohttp_client_session.ws_connect(
  File ".../aiohttp/client.py", line 1244, in _ws_connect
    resp = await self.request(
  File ".../aiohttp/client.py", line 581, in _request
    raise RuntimeError("Session is closed")
RuntimeError: Session is closed

How the loop becomes orphaned

close() (lines 446–457) cancels current_session_monitor and message_receiver fire-and-forget — it calls .cancel() but does not await the tasks — and then immediately closes the ClientSession. Both tasks are created via asyncio.ensure_future as independent top-level tasks, and both can call connect_to_new_endpoint() (the monitor on staleness, the receiver on a CLOSE frame). If one of them is inside a reconnect attempt when the cancel is requested, its in-flight connect() call keeps running against the now-closed session. At that point nothing references the task anymore: the owning application has replaced its handler/client, but the retry loop lives on.

Observed in production (three long-running gateways):

  • exact 10.000s cadence (= default ping_interval), never accelerating or backing off;
  • one host accumulated 270,136 RuntimeError: Session is closed events over 46 days;
  • another host showed two parallel orphaned loops (event pairs 10–13ms apart, both at 10s cadence, stable for 1.5h+);
  • the count of parallel loops never changes once established — consistent with rare races each leaking one loop;
  • only a full process restart clears it. An application-level watchdog (which rebuilds the handler) cannot, because the orphaned task is unreachable.

Suggested fixes (any one of these would break the failure mode)

  1. In connect()'s exception handler: if self.closed (or self.aiohttp_client_session.closed), log once and return instead of retrying.
  2. At the top of connect(): recreate self.aiohttp_client_session when it is closed (mirrors what the Node.js @slack/socket-mode SDK effectively does by building a fresh connection per attempt).
  3. In close(): after cancelling current_session_monitor / message_processor / message_receiver, await asyncio.wait([...], timeout=...) so no cancelled task is still mid-connect when the session closes.

Workaround we deployed

Application-side: cancel and await client.current_session_monitor / client.message_processor / client.message_receiver before calling close_async(), plus a logging-filter-based storm detector that restarts the process (supervisor restarts it) if the "Failed to connect (error: Session is closed)" cadence is detected anyway. Happy to turn any of the suggested fixes into a PR if maintainers indicate a preferred direction.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:asyncbugM-T: A confirmed bug report. Issues are confirmed when the reproduction steps are documented

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions