Skip to content

Wait for durable redeemer handoff before runner retirement - #87

Closed
j0sh wants to merge 3 commits into
mainfrom
ja-codex/durable-handoff-drain
Closed

j0sh wants to merge 3 commits into
mainfrom
ja-codex/durable-handoff-drain

Conversation

@j0sh

@j0sh j0sh commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Runner unregister returns 503 until external redeemers durably acknowledge outstanding winning tickets, including after registration loss or restart. Receipts are fenced during removal; destination changes use the same handoff boundary while retaining signed-nonce checks.

Validation: full repository race suite with pinned Go and Python SDK fixtures, go vet, and Staticcheck v0.8.2.

Required by livepeer/control-center#3 for livepeer/naap-ops#119.

@j0sh
j0sh deployed to docker-publish November 12, 2026 16:42 — with GitHub Actions Active
@j0sh
j0sh deployed to docker-publish November 12, 2026 16:43 — with GitHub Actions Active
@j0sh
j0sh deployed to docker-publish November 12, 2026 16:43 — with GitHub Actions Active
@j0sh
j0sh deployed to docker-publish November 12, 2026 16:43 — with GitHub Actions Active
@j0sh
j0sh deployed to docker-publish November 12, 2026 17:12 — with GitHub Actions Active
@j0sh
j0sh deployed to docker-publish November 12, 2026 17:13 — with GitHub Actions Active
@j0sh
j0sh deployed to docker-publish November 12, 2026 17:13 — with GitHub Actions Active
@j0sh
j0sh deployed to docker-publish November 12, 2026 17:13 — with GitHub Actions Active
@j0sh
j0sh marked this pull request as ready for review November 12, 2026 17:14
Copilot AI balanced review requested due to automatic review settings November 12, 2026 17:14

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unregister requests can exceed the intended deadline while waiting on the payment mutex, and absent-runner requests can access shared payment state without authentication.

2 open issues
What changed in this PR

This PR adds a durable ticket-handoff boundary to runner retirement and redemption destination changes, supporting managed paid-orchestrator lifecycle operations.

Changes:

  • Return 503 while external ticket handoff is pending or unavailable, including after registration loss.
  • Fence receipts from removed app registrations to prevent late ticket commits.
  • Allow destination changes after durable acknowledgement, without waiting for settlement, while retaining signed-nonce checks.
File Description
redemption/​state.go Adds handoff-drain checks and updates destination-change eligibility.
redemption/​http_test.go Tests acknowledgement, restart persistence, and destination changes.
orchestrator/​registry.go Fences removed registrations and checks handoff before completing unregister.
orchestrator/​payment_engine.go Rejects receipts tied to retired app registrations.
orchestrator/​drain_test.go Tests retirement retries and receipt fencing during validation.

Review effort: Balanced

Comment thread orchestrator/registry.go
r.mu.Lock()
defer r.mu.Unlock()
current := r.runners[id]
if current != nil && (current.Static || !equalSecret(current.Credential, auth)) {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed with Astra. We’re keeping this behavior in this PR to preserve retirement retries after removal, expiry, or restart, when the scoped credential is no longer available to authenticate against. Requiring bootstrap authentication would reject retries using the original scoped credential; retaining and authenticating it across restart needs a separate recovery protocol.

The absent-runner path reports coarse busy/clear status and cannot remove a live runner without its credential. The mutex-wait issue is fixed in 386aa8f.

Comment thread orchestrator/registry.go Resolved
Copilot AI balanced review requested due to automatic review settings November 12, 2026 17:31
@j0sh
j0sh deployed to docker-publish November 12, 2026 17:34 — with GitHub Actions Active

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are identified for the changes gating runner retirement on durable ticket acknowledgement and fencing late payment receipts.

1 open issue
1 resolved since last review

Review effort: Balanced

@j0sh
j0sh deployed to docker-publish November 12, 2026 17:35 — with GitHub Actions Active
@j0sh
j0sh deployed to docker-publish November 12, 2026 17:35 — with GitHub Actions Active
@j0sh
j0sh deployed to docker-publish November 12, 2026 17:36 — with GitHub Actions Active
@j0sh

j0sh commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

Closing unmerged after reviewing the requirements with Astra and coordinating with livepeer/control-center#3. Runner unregister was incorrectly used to gate Node-wide payment handoff. Existing durable storage and retry behavior support ordinary orchestrator replacement with retained state, so this PR is unnecessary for that workflow.

Control Center is removing the synthetic unregister/payment probe and dependency on this PR. Disabling paid processing or changing retained-state ownership requires an explicit configuration-transition policy; it must not be inferred from runner retirement.

@j0sh j0sh closed this Nov 12, 2026
@j0sh
j0sh deleted the ja-codex/durable-handoff-drain branch November 12, 2026 18:17

This branch was successfully deployed

1 active deployment
docker-publish — 386aa8f6 Deployed Nov 12, 2026 by j0sh via docker-pr / promote #181
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants