Skip to content

Retry fenced workflow task heartbeats after backend loss - #79

Merged
rmcdaniel merged 1 commit into
mainfrom
fix/worker-heartbeat-backend-recovery
Sep 26, 2026
Merged

rmcdaniel merged 1 commit into
mainfrom
fix/worker-heartbeat-backend-recovery

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Problem

Server issue durable-workflow/server#137 exposed an unhandled workflow-task heartbeat backend failure during a MySQL interruption. The matching Server change returns a fenced backend_unavailable 503.

Change

  • Recognize that typed worker heartbeat response at both root and path-prefixed managed runtime URLs.
  • Retry only when the response repeats the submitted task ID, lease owner, attempt, worker identity, and valid retry metadata.
  • Fail closed for a mismatched or malformed response.

Verification

  • test_storage_admission.py: 94 tests passed in Python 3.12.
  • Ruff and MyPy passed for the changed code.

Published SDK and Server artifacts still need the MySQL interruption retest before issue #137 can qualify.

@rmcdaniel
rmcdaniel marked this pull request as ready for review September 26, 2026 15:25
@rmcdaniel
rmcdaniel merged commit acbbc04 into main Sep 26, 2026
13 checks passed
@rmcdaniel
rmcdaniel deleted the fix/worker-heartbeat-backend-recovery branch September 26, 2026 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants