Skip to content

Guard worker build and task queue retirement against live dependencies #519

Open

Description

@rmcdaniel

User problem and decision

Promotion does not make old pinned runs or recorded activity queues independent of the old fleet. An operator needs a supported answer to “can this build or queue be retired safely?” and a guarded action, not only a warning to manually keep workers forever.

This is the product capability behind durable-workflow/durable-workflow.github.io#262, #274 and #308. Retain their accurate current promotion/routing documentation; that does not deliver safe-retirement enforcement.

At DeploymentLifecycleService::drain, Core's planner receives no pinned-run/recorded-queue dependency inventory. DeploymentLifecyclePlan::evaluateDrain can permit an active deployment without proving those dependencies are gone. #317 already delivered routing diagnostics; it did not prevent unsafe retirement.

Acceptance

  • Expose a bounded, namespace/queue/build-scoped retirement inspection covering passive open pinned runs, continue-as-new chains, children, recorded activity queues, delayed retries and pending/leased tasks. An empty ready-task count is insufficient.
  • Add an explicit guarded retirement operation. Preserve the separate emergency-drain contract deliberately; do not silently change an existing operational action without compatibility review.
  • Refuse unsafe retirement with concrete blocker identities/counts, observation completeness and a safe recovery action. Missing/stale/incomplete inventory is unavailable, not permission to retire.
  • Make the check/action safe against concurrent starts, task creation, retries and continuation. A non-atomic zero-count preview is not an enforcement barrier.
  • Preserve recorded run build pins, activity queues, original history and deadlines. Do not automatically re-pin or reroute existing work. Compatible upgrades and any future migration API need their own explicit policy and tests; coordinate Changed Python workflow definitions silently strand in-flight runs #518.
  • Prove a parked old-build run and a delayed old-queue activity retry block retirement even while the promoted new fleet is healthy. Then complete/drain their real dependencies and prove retirement succeeds. Test isolation, cancellation, restart and continued chains.
  • Expose the workflow through CLI/SDK/operator views and publish the actual promotion, retirement and recovery journey. Keep existing BuildId filters useful as manual inspection without misrepresenting them as the guarded action.

Priority P2, additive operations-safety feature. General owns PHP/CLI/package implementation after an agreed source window. Native/shared adoption remains separately qualified under #325; frozen cohorts do not change through this issue.

Activity

  1. added
    kind:featureA public product capability or experience is requested
    priority:P2Normal-priority product work
    status:readyReady for implementation
    on Oct 20, 2026
  2. rmcdaniel commented on Nov 4, 2026

    @rmcdaniel
    MemberAuthor

    Server open-item pass, 2026-11-04.

    Owner: Server for native bounded retirement inspection/guard, Workflow for definition/marker provenance, CLI/Waterline for consumers. Central220’s branch-use question belongs here with #325, not in a new competing retirement tracker. No current native endpoint is identified that proves open runs are independent of a change_id/version.

    Next source contract distinguishes recorded legacy markers from unknown/unreached runs on older or missing definitions, includes passive pinned runs/CAN/children/recorded queues/delayed retries and binds guarded retirement to inventory revision. Absence of a marker or ready task is never proof of safe removal. Workflow: return canonical admission/definition provenance; General reviews the unknown-state policy and source order before endpoint/consumer implementation.

  3. rmcdaniel commented on Nov 4, 2026

    @rmcdaniel
    MemberAuthor

    Core provenance fixture pointer requested in Server325/6643408164: public 2.20.3 WorkflowDefinitionFingerprint and FailedRunRedrivePlan bind declared definition and recorded marker/reuse history. In V2FailedRunRedriveTest, testServiceRunRequiresMatchingRecordedWorkerDefinitionFingerprint, testADefinitionThatDropsTheRecordedVersionNamesTheMarkerStep, testRedriveReusesOneMarkerForARepeatedPatchCall, testALaterRedriveOnTheSameDefinitionRecordsNoInheritedChange and testRedriveRefusesABrokenReuseProvenancePairWithoutMutation are the concrete positive/refusal fixtures. Marker absence on an unreached older definition is not a proven safe-cleanup zero; that separate visibility contract remains tracked in .github220/6642217365. Source lineage and a closed issue are not an installed native marker/definition receipt. Server owns that protocol/endpoint map before any native producer change.

    Delivery evidence remains the exact public Core 2.20.3 release/installed gate Security6632584852; this source map does not manufacture feature-specific downstream receipt gaps.

  4. rmcdaniel commented on Nov 11, 2026

    @rmcdaniel
    MemberAuthor

    Triage 2026-11-11: still open; drain has no pinned-run inventory. Backlog feature. Owner: server.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent:serverkind:featureA public product capability or experience is requestedpriority:P2Normal-priority product workstatus:readyReady for implementation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions