Skip to content

Re-sign each retry [RHELDST-47596, RHELDST-47601] - #40

Open
crungehottman wants to merge 1 commit into
release-engineering:masterfrom
crungehottman:resign-each-retry
Open

crungehottman wants to merge 1 commit into
release-engineering:masterfrom
crungehottman:resign-each-retry

Conversation

@crungehottman

Copy link
Copy Markdown
Member

Retries were performed by urllib3's Retry on the HTTPAdapter, which resends the request below the requests auth layer -- so EdgeGridAuth signed once and every retry reused the same Authorization header. Once a request's signature aged past Akamai's ~30s timestamp tolerance, every retry was rejected with 400 "Invalid timestamp", regardless of whether the original rate limiting had cleared, until the retry budget was exhausted and the commit failed.

Replace the transport-level retry with an application-level retry loop in __start_purge: each attempt re-issues session.post(), so EdgeGridAuth re-signs with a current timestamp and a retry can succeed once the rate limit clears.

  • compute_backoff() mirrors urllib3's exponential backoff (first retry 0, then factor2**(n-1) + jitter, capped), so the FAST_PURGE_RETRY_ tunables keep their meaning.
  • Retryable statuses are an explicit allowlist (anything else fails fast), configurable via FAST_PURGE_RETRY_STATUSES. The default covers 429, all 5xx, and the EdgeGrid-specific 400 (returned for a stale timestamp and for WAF rate-control, both of which recover on a fresh attempt).
  • Connection/timeout errors are retried.
  • Failed attempts now log the response body and Retry-After, so the actual Akamai error (e.g. "Invalid timestamp") is visible.
  • A non-JSON 201 is tolerated rather than raising.

Retries were performed by urllib3's Retry on the HTTPAdapter, which
resends the request below the requests auth layer -- so EdgeGridAuth
signed once and every retry reused the same Authorization header. Once a
request's signature aged past Akamai's ~30s timestamp tolerance, every
retry was rejected with 400 "Invalid timestamp", regardless of whether
the original rate limiting had cleared, until the retry budget was
exhausted and the commit failed.

Replace the transport-level retry with an application-level retry loop in
__start_purge: each attempt re-issues session.post(), so EdgeGridAuth
re-signs with a current timestamp and a retry can succeed once the rate
limit clears.

- compute_backoff() mirrors urllib3's exponential backoff (first retry 0,
  then factor*2**(n-1) + jitter, capped), so the FAST_PURGE_RETRY_*
  tunables keep their meaning.
- Retryable statuses are an explicit allowlist (anything else fails fast),
  configurable via FAST_PURGE_RETRY_STATUSES. The default covers 429, all
  5xx, and the EdgeGrid-specific 400 (returned for a stale timestamp and
  for WAF rate-control, both of which recover on a fresh attempt).
- Connection/timeout errors are retried.
- Failed attempts now log the response body and Retry-After, so the
  actual Akamai error (e.g. "Invalid timestamp") is visible.
- A non-JSON 201 is tolerated rather than raising.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (5abf963) to head (5d7913f).

Additional details and impacted files
@@            Coverage Diff            @@
##            master       #40   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            2         2           
  Lines          152       174   +22     
=========================================
+ Hits           152       174   +22     
Flag Coverage Δ
unit-tests 100.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants