Harden Resque post-fork reconnect against saturated Redis (maxclients) [179D-310]

[brianlball]

Cherry-pick of #873 onto the 179D-310 test branch (clean pick of f13e0c8c, no conflicts), so the fix rides the 3.10.0-179D-test images.

Problem

During the 2026-08-18 k8s deploy testing on the 179D-310 images, every analysis_wrappers job popped between 13:44 and 13:55 UTC vanished without a trace, stranding six analyses in queued forever (and losing three FinalizeAnalysis runs). The resque log showed:

(Job{analysis_wrappers} | ResqueJobs::InitializeAnalysis | [...]) failed: #<Redis::CommandError: ERR max number of clients reached>
Received exception when reporting failure: #<Redis::CommandError: ERR max number of clients reached>
Received exception when increasing failed jobs counter (redis issue) : #<Redis::CommandError: ERR max number of clients reached>

Root cause chain:

1. A large spot-instance worker fleet (plus connections orphaned by unwarned node reclaims) pushed redis connected_clients past maxclients.
2. Resque forks a child per job; the child must open a new redis connection (Resque::Worker#reconnect). Stock resque 2.6 only retries Redis::BaseConnectionError there — but a saturated server accepts the socket and replies -ERR max number of clients reached, which redis-rb raises as Redis::CommandError. No retry: the child dies ~4ms after got:.
3. Reporting the failure needs redis too, so job.fail and the failed counter also throw — the job is never recorded anywhere, and the analysis sits in queued with no terminal state.

The capacity side is being fixed in the helm chart (redis.maxclients, keepalive/timeout). This PR is the app-side hardening so a brief saturation window degrades to a short wait instead of silent job loss.

Change

• server/config/initializers/resque_reconnect_retry.rb — prepends an override of Resque::Worker#reconnect that also retries Redis::CommandError (covers maxclients saturation and -LOADING during AOF replay after a redis restart), 5 tries with increasing backoff (~30s total), then re-raises into the stock failure path. No-op in delayed_job deployments; retry count tunable via RESQUE_RECONNECT_RETRIES.
• server/spec/lib/resque_reconnect_retry_spec.rb — unit specs, tagged depends_resque.
• docker/server/run-server-tests.sh — runs the new spec in the docker (resque) CI job.

Testing

Verified in a local replica of the CI docker environment using the 3.10 image lineage of this branch (nrel/openstudio-server:3.10.0 + mongo:8.0.12 as db + redis:6.0.9 as queue, RAILS_ENV=docker): 6 examples, 0 failures. Same result on the develop image (see #873).

🤖 Generated with Claude Code