Harden Resque post-fork reconnect against saturated Redis (maxclients)

[brianlball]

Problem

During the 2026-08-18 k8s deploy testing, every analysis_wrappers job popped between 13:44 and 13:55 UTC vanished without a trace, stranding six analyses in queued forever (and losing three FinalizeAnalysis runs). The resque log showed:

(Job{analysis_wrappers} | ResqueJobs::InitializeAnalysis | [...]) failed: #<Redis::CommandError: ERR max number of clients reached>
Received exception when reporting failure: #<Redis::CommandError: ERR max number of clients reached>
Received exception when increasing failed jobs counter (redis issue) : #<Redis::CommandError: ERR max number of clients reached>

Root cause chain:

1. A large spot-instance worker fleet (plus connections orphaned by unwarned node reclaims) pushed redis connected_clients past maxclients.
2. Resque forks a child per job; the child must open a new redis connection (Resque::Worker#reconnect). Stock resque 2.6 only retries Redis::BaseConnectionError there — but a saturated server accepts the socket and replies -ERR max number of clients reached, which redis-rb raises as Redis::CommandError. No retry: the child dies ~4ms after got:.
3. Reporting the failure needs redis too, so job.fail and the failed counter also throw — the job is never recorded anywhere, the app rescue in InitializeAnalysis never runs (the job body never started), and the analysis sits in queued with no terminal state.

The capacity side is being fixed in the helm chart (redis.maxclients, keepalive/timeout). This PR is the app-side hardening so a brief saturation window degrades to a short wait instead of silent job loss.

Change

• server/config/initializers/resque_reconnect_retry.rb — prepends an override of Resque::Worker#reconnect that also retries Redis::CommandError (covers maxclients saturation and -LOADING during AOF replay after a redis restart), 5 tries with increasing backoff (~30s total), then re-raises into the stock failure path. During reconnect the only commands on the wire are connection setup, so a CommandError there is effectively connection-level and safe to retry. No-op in delayed_job deployments (guarded on defined?(Resque::Worker)); retry count tunable via RESQUE_RECONNECT_RETRIES.
• server/spec/lib/resque_reconnect_retry_spec.rb — unit specs: override active, success path, CommandError retried with backoff, BaseConnectionError behavior preserved, re-raise after exhausting retries, unrelated errors not swallowed. Tagged depends_resque since resque only loads in resque envs.
• docker/server/run-server-tests.sh — runs the new spec file in the docker (resque) CI job, alongside the other fast no-stack specs.

Testing

Verified in a local replica of the CI docker environment (nrel/openstudio-server:develop + mongo:8.0.12 as db + redis:6.0.9 as queue, RAILS_ENV=docker):

ResqueReconnectRetry
  overrides Resque::Worker#reconnect
  reconnects once and does not sleep when the connection succeeds
  re-raises after exhausting the retries
  still retries Redis::BaseConnectionError (stock resque behavior preserved)
  retries Redis::CommandError (e.g. maxclients saturation) with backoff until it succeeds
  does not swallow or retry unrelated errors

6 examples, 0 failures

🤖 Generated with Claude Code