Joebecher/ccmrg 2042 cap bundle analysis processor at 10 attempts and fix

[drazisil-codecov]

bump ci

---

Note

Medium Risk
Changes core retry/lock-acquisition control flow and error handling for multiple Celery tasks, which can alter when tasks stop retrying and how failures are acknowledged. While well-covered by updated tests, misconfiguration or edge cases could prematurely stop processing or change chain outcomes.

Overview
Aligns retry semantics across workers so max_retries consistently means maximum total attempts (stop when attempts >= max_retries), updating BaseCodecovTask._has_exceeded_max_attempts/safe_retry and related config comments.

Enhances LockManager to track lock-acquisition attempts in Redis (lock_attempts:* with 1-day TTL), raise LockRetry(max_retries_exceeded=True) when the counter hits the cap, and clear the counter after a successful lock; tasks (bundle_analysis_processor, manual_trigger, preprocess_upload, upload_finisher, and notification/finisher tasks) now honor this cap to avoid infinite retry loops on lock contention or message re-delivery.

Refactors bundle_analysis_processor error paths to log consistently and to set upload state to error (best-effort commit) while returning previous_result instead of re-raising in several failure cases, preserving Celery chain behavior; updates unit/integration tests accordingly (notably mocking redis.incr and asserting the new attempt-based boundaries/log fields).

^{Written by Cursor Bugbot for commit 1fe32c5. This will update automatically on new commits. Configure here.}

[linear]

CCMRG-2042 Cap bundle analysis processor at 10 attempts and fix infinite retry

[sentry]

Bug: The Redis INCR and EXPIRE commands for the lock attempt counter are not atomic. A failure between these calls can create a counter with no TTL, permanently blocking the lock.
_{Severity: HIGH}

Suggested Fix

To ensure atomicity, use a Redis transaction (MULTI/EXEC) or a Lua script to perform the INCR and EXPIRE operations together. This guarantees that either both commands succeed or neither does, preventing the creation of a counter key without a TTL.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location: apps/worker/services/lock_manager.py#L179-L181

Potential issue: When lock acquisition fails, the code increments a Redis counter
(`INCR`) and then sets a TTL on it (`EXPIRE`). These are two separate, non-atomic
operations. If the `INCR` call succeeds but the `EXPIRE` call fails (e.g., due to a
Redis timeout or connection error), the counter key will be created without a TTL. If
this counter reaches the maximum retry limit, it will never be cleared, as clearing only
happens on successful lock acquisition. This will permanently block any future attempts
to acquire the same lock, effectively poisoning it.

_{Did we get this right? 👍 / 👎 to inform future reviews.}

[codspeed-hq]

Merging this PR will not alter performance

✅ 9 untouched benchmarks

---

_{Comparing joebecher/ccmrg-2042-cap-bundle-analysis-processor-at-10-attempts-and-fix (1fe32c5) with main (afa4356)¹}

Footnotes

1. No successful run was found on main (9b95e6b) during the generation of this report, so afa4356 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩