Skip to content

Prevent update-validator polling from delaying ordinary workflow tasks #31

Description

@rmcdaniel

Problem

The Python Worker now correctly bounds synchronous update-validation work with the workflow-task semaphore. However, both the ordinary workflow poller and update-validation poller acquire that shared semaphore before entering an HTTP long poll. With max_concurrent_workflow_tasks=1, an empty poll for either task kind can occupy the only reservation for up to the configured long-poll timeout (35 seconds by default), delaying discovery of ready work on the other endpoint. Heartbeat availability is derived from executing tasks, so it can also report a free workflow slot while an empty long poll owns the admission token.

This is a latency and observability defect, not a reason to remove bounded validator admission.

Acceptance criteria

  • Preserve a hard bound on leased and executing workflow/update-validation work; a burst of validator tasks cannot accumulate unbounded replay work.
  • Ordinary workflow task discovery and update-validation discovery do not serialize behind an unrelated empty long poll. A worker configured with one workflow slot must promptly discover either task kind when the other endpoint is idle.
  • Worker heartbeat slot reporting matches the chosen admission model, including reservations and leased work.
  • Avoid a busy-poll loop or a material increase in idle request load. If the correct design requires a Server endpoint that multiplexes workflow and validation task kinds, make that a deliberate cross-repository contract instead of hiding latency in the SDK.
  • Add deterministic tests with one workflow slot, both pollers active, one endpoint held empty, and a task ready on the other endpoint. Cover both directions, shutdown cancellation, capacity release, and bounded dispatch latency without wall-clock sleeps.
  • Publish the next Python prerelease and require focused worker tests plus current workflow-updates conformance before closing.

Evidence

The capacity implementation landed in Python RC23 source at fb48695b9a5a7dfc9576f24b9aa3f10bf15eec8e. _poll_workflow_tasks() and _poll_update_validation_tasks() both acquire _wf_semaphore before their long-poll request; the new tests prove the concurrency ceiling but do not run both pollers together or test cross-kind latency.

Metadata

Metadata

Assignees

No one assigned

    Labels

    authority:githubGitHub is the authoritative lifecycle record for this workbeta:blockerMust be resolved before the first 2.0 betacompletion:evidence-requiredClose only after all explicit acceptance and operational evidence is publickind:defectA public product behavior is incorrectpriority:P1High-priority product or release riskrepo:sdk-pythonOwned by the Python SDK repositorystatus:readyReady for implementation

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions