RedisWorker leaks Redis locks in multiple code paths

[dkliban]

Problem

Redis locks in pulpcore have no TTL and persist forever. There are 6 code paths where locks can be acquired but never released, causing resources to be permanently blocked. During the 2026-07-24 incident, a single dead worker left 543 orphaned task locks and 524 orphaned resource locks that blocked the entire task system for ~23 hours.

Leak Paths

Path 1: fetch_task() exception handler (patch exists)

In RedisWorker.fetch_task(), when acquire_locks() succeeds but a subsequent operation throws an exception, the except handler does continue without releasing locks.

Fix: Add safe_release_task_locks(task, lock_owner=self.name) to the except handler.

Path 2: supervise_task() subprocess killed by signal

When a subprocess is OOMKilled after writing task state to DB but before its finally block releases Redis locks, the parent sees the task in a final state at line 636 and skips _maybe_release_locks:

if task.state not in TASK_FINAL_STATES:
    self._maybe_release_locks(task)  # skipped if task already completed/failed

Fix: Move _maybe_release_locks(task) to a finally block so it runs unconditionally regardless of task state.

Path 3: supervise_task() cancel path refresh_from_db() fails

If task.refresh_from_db() at line 633 throws a DatabaseError (DB contention), the entire cancel block is skipped including lock release.

Fix: Wrap the cancel block in try/except/finally with lock release in the finally.

Path 4: handle_tasks() no finally block for deferred tasks

The finally block at line 675-681 only releases locks for immediate tasks:

if task and task.immediate:
    self._maybe_release_locks(task)

If supervise_task() throws an unhandled exception for a deferred task, locks are never released.

Fix: Change if task and task.immediate: to if task:. Safe because _maybe_release_locks is idempotent.

Path 5: cancel_task() doesn't release Redis locks for waiting tasks

When canceling a WAITING task (app_lock is None), the code sets app_lock and calls set_canceled() but never releases Redis locks. Race condition: a worker may have acquired Redis locks but not yet set app_lock.

Fix: Add a helper that reads the actual lock owner from the Redis task lock key, then releases all associated resource locks. Call after set_canceled() in the waiting-task path.

Path 6: dispatch() immediate task lock release fails

If safe_release_task_locks() fails because Redis is transiently unavailable, the exception re-raises and locks remain forever.

Fix: Wrap with a 3-attempt retry loop with backoff.

Proposed Implementation

All fixes rely on the idempotency of _maybe_release_locks (checks _all_locks_released flag) and the Lua release script (verifies ownership before deleting). Double-calling is a no-op.

Files to modify

• pulpcore/tasking/redis_worker.py — paths 1-4
• pulpcore/tasking/redis_tasks.py — paths 5-6

References

• RCA: 2026-07-24/25 tasking degradation
• Supersedes: supervise_task() should release locks unconditionally regardless of task state #7915, handle_tasks() finally block should release locks for all task types, not just immediate #7916, cancel_task() should release Redis locks when canceling waiting tasks #7917, dispatch() should retry lock release on transient Redis failure #7918