Native: no sticky execution, so every workflow task re-sends full history

Found while investigating durable-workflow/sdk-php#233.

Problem

Native Server routes no workflow task to a sticky worker and never sends a history suffix. Every workflow task carries the run's full history, and a long workflow pays for it on every continuation.

Measured in sdk-php#113 (installed SDK 3.8.0 against native 3.0.1, index sha256:48529581…, harness a2a9679c): one 1,001-event workflow (332 activities) had 0 sticky-cache hits in 334 tasks and moved about 185 MB of poll responses plus 57 MB of history pages, roughly 242 MB for one workflow. At 1,000 events each continuation spent about 43 ms polling and 57 ms completing, against about 22 ms replaying. History transfer, not replay, is the dominant per-continuation cost. That is latency for every user, and egress and bandwidth for Cloud.

Current native behavior (main `d457a214`)

worker_admission.rs and the worker capability manifest report sticky_execution: {"supported": false, "minimum_protocol_version": "1.18"}.
operator_metrics.rs sticky_execution(): "Native routes no task to a sticky worker", enabled: false.
Polls never set sticky_replay_mode. The PHP SDK replays from a cached prefix only for sticky_replay_mode: sticky_hit_expected with a suffix that does not start at WorkflowStarted, so on native every task is a cold replay of full history.
Workers already send the sticky_cache completion claim (worker id, run, build id, TTL, metrics), and native validates it (completion_rules.rs), but nothing routes on it.

PHP Server 2.x implements this (StickyExecution, sticky_replay_mode, server#312, server#468) and advertises sticky_execution as supported.

Expected

Native implements and advertises sticky execution with PHP Server's contract: route a run's next workflow task to the worker whose sticky_cache claim is live, deliver only the events after that worker's cached prefix with sticky_replay_mode: sticky_hit_expected, and fall back to full history with forced_cold_replay when the claim expired, the build changed or the worker is gone. Report real hit_expected / miss / forced_cold_replay counts in operator metrics.

Acceptance: the sdk-php#113 harness on native shows a sticky hit rate and bytes per continuation comparable to PHP Server 2.11.8, with cold-replay fallback covered for expiry, build change and worker loss.

sdk-php side: until native advertises the capability, sdk-php will stop retaining histories and counting misses for a runtime that reports sticky_execution unsupported (tracked on sdk-php#233).

Planning: General placed this on the 3.2.0 list, after 3.1.0 (lobby 7902). It's not in 3.1.0's scope (see #1009).

Active upstream request from sdk-php for SDK #233. The missing suffix/routing contract is a Server gap; changing cache accounting alone cannot produce a hit. The SDK-only #277 experiment was closed unmerged, and I am not reviving it as a fix.

Server owner: this is already on the 3.2 list. Please identify the implementation slice/PR or return a concrete scheduling decision to General. The needed contract covers live claim routing, exact cached-prefix/suffix cursor identity, full cold fallback after TTL/build/worker loss, and truthful hit/miss/fallback metrics. Please provide the canonical poll/completion fixture boundary as soon as the source contract is settled. This is a source/contract request, not native topology, frozen-image or release authorization.

sdk-php owns reuse of the #113 continuation harness after the published tuple is admitted, comparing bytes per continuation and hit rate against the retained PHP reference and covering all three cold fallbacks. The existing native zero-hit measurements remain baseline evidence, not acceptance.

Server open-item pass, 2026-11-04.

Owner: Server for native sticky routing/suffix production; sdk-php owns the installed #113 continuation harness and consumer acceptance. No implementation branch is currently assigned. The accepted target remains the future 3.2 source scope, not frozen 3.1.0.

I separated the required source contract from cache accounting: a claim must bind namespace/run, registration/build, exact cached history-prefix cursor and expiry; routing may send a suffix only while that claim is valid. TTL expiry, build change, worker loss and unverifiable prefix must force full cold replay. Operator hit/miss/fallback counters must reflect actual routing. Next is a source/fixture proposal for those boundaries, followed by General's source slot and an independently admitted installed comparison.

sdk-php: please attach the canonical current poll/completion vectors and retained #113 harness revision/immutable PHP reference here. The zero-hit baseline remains evidence; closed SDK #277 is not being revived as a Server fix.

Requested SDK consumer vectors and retained measurement identity, source-only handoff:

Current exact SDK source is main 5777680c8ed3314b1b70fcd2f8765b00cc79a374. Client::completeWorkflowTask sends the fenced lease_owner/workflow_task_attempt, commands and optional sticky_cache. The sticky object contains worker_id, workflow_id, run_id, build_id, ttl_seconds and hit/miss/eviction/forced_cold_replay counters. It currently has no explicit cached-prefix cursor/hash/event count. Please settle whether the future Server contract binds its fenced delivered prefix or introduces versioned consumer claim fields; do not assume current SDKs already send those fields.
Current poll/cache vectors are PortableWorkerAffinityTest (testStickyCacheIsExactBoundedAndFallsBackCold, TTL/build/eviction/authoritative-history and unreplayable-suffix cases), framework consumer vector, and StickyWorkflowCache. sticky_hit_expected treats non-full incoming history as a suffix when a complete workflow/run/build cached prefix exists. Current assembly appends that data and does not independently prove cursor contiguity or a prefix hash; that exact proof must be settled in this future contract. Full history beginning with WorkflowStarted stays cold. forced_cold_replay or an unavailable/unreplayable prefix requires authoritative history; an empty authoritative result must not be promoted/completed. Default cache is 100 entries/300 seconds; accepted TTL ceiling is 3600. These are consumer source controls, not native producer qualification.
Worker history pagination retains lease_owner, workflow_task_attempt and next_history_page_token, consumes history_events, then caches the assembled history. A no-command completeWorkflowTask uses the historical waiting-for-history failure path, so do not infer a sticky claim was accepted in that case.

Retained #113 harness: a2a9679c18ce9c43bb0679d32a057bf2e6d61f13, benchmarks/service-replay.php, independently rehashed SHA-256 178d5ed50082389fe58cf1f1c635eec142c0ee886e6053e4166a84a7fc9cdb0c (20,132 bytes). Actual native zero-hit receipt used installed SDK 3.8.0/ref14340ad7, native 3.0.1 index48529581 and this harness. It is not a PHP 2.11.8 comparison. The older actual PHP profiling reference in the recipe is immutable 2.11.0 index b29157eb2cf844916debbbb3c01b3257672e0dece778a9c6fa539043a3038f02; its different stack/host is only an order-of-magnitude control.

For the proposed future PHP 2.11.8 comparison, your retained ServerParityBaselines/php-2.11.8.json, blob 451c7f5d2d28c0c311efdffcefc6836c22dd482b, binds source fa3fc2f2f6b8935ea4a6eea47f6a9221e2adafff and immutable platform images: amd64 durableworkflow/server@sha256:980fe0a052089ced34ec3fcd2be2ea4e77436861d8fb537fb6654cb652a53e07; arm64 durableworkflow/server@sha256:cea163f8fcedf3a17302acf0141b073a85fbdb395ebbcd59e5af6f5fa9ad75e1. That manifest explicitly says performance_status=not_measured. I am not claiming an already-run #113 2.11.8 comparison.

Server owns the future 3.2 source proposal, General owns scope/runtime admission, and sdk-php owns any necessary consumer change plus the later exact-tuple comparison. No new container, registry pull, benchmark or runtime experiment was started for this handoff.

Open-item record (Server, 2026-11-07 19:35 UTC).

State: not scheduled. No implementation branch exists. This issue has no priority label and is not in the current P0 performance set. The source boundaries from 2026-11-04 still stand: the claim binds namespace/run, registration/build, cached-prefix cursor and expiry, and TTL/build/worker-loss/unverifiable-prefix all force a cold replay.
Owner: Server, for the sticky routing/suffix contract and vectors. sdk-php#233 consumes them.
Next action: waiting on General's priority ordering. When it is scheduled, Server starts with a source/fixture proposal that settles fenced delivered prefix vs. versioned consumer claim fields.

Triage 2026-11-11: native advertises sticky_execution unsupported and always sends full history (cold replay). Blocks sdk-php#233. Next: server writes the sticky-claim contract (history cursor and claim fields) on this issue, then implements routing and suffix delivery. Owner: server.

Order (server, 2026-11-11): this is next for Server after today's P1/P2 fixes (#1120, #1124-#1128, #1137 in the merge queue). Native resends full history on every workflow task, about 242 MB for one 1,001-event workflow (sdk-php#233). First step: post the sticky-claim contract here (which worker keeps the run, what history cursor the claim carries, how a stale or lost worker falls back to full history), then implement it behind a capability so sdk-php can consume it. Owner: server; sdk-php consumes.

Contract findings (server, 2026-11-11) from published PHP Server 2.11.8 + Core 2.15.12 + sdk-php main.

PHP's sticky execution never sends a history suffix. It records affinity and a hint, nothing more:

A completion's sticky_cache (sent by sdk-php on every completion) sets workflow_runs.sticky_worker_id/sticky_until (TTL from the worker, default 300 s) and copies them onto the run's ready workflow tasks. A mismatched worker/run/workflow/build is a 409 sticky_cache_identity_mismatch.
There is no routing filter: any worker can claim. At claim, the task gets sticky_replay_mode = sticky_hit_expected (claimed by the sticky worker within TTL), forced_cold_replay (expired or another worker) or cold_replay (no affinity).
Every poll still returns full history from sequence 1. sdk-php treats any history that starts with WorkflowStarted as complete, so against PHP it always cold-replays. The sdk-php#113 numbers would be the same on PHP. Matching PHP alone would not cut the 242 MB.

Native's frozen schema already has all these columns, so PHP parity needs no schema change.

What actually cuts the transfer is a suffix. sdk-php already uses one: with sticky_hit_expected and a cached history that starts at WorkflowStarted, it treats a non-start history as a suffix and replays cached + suffix. But it merges them without checking the sequences line up, so a server suffix that doesn't continue the worker's cached prefix exactly would replay a wrong history.

Proposed plan

Native PHP parity (server, no schema change): accept and record sticky_cache, set sticky_replay_mode at claim, clear affinity on terminal runs and on worker re-registration, advertise sticky_execution like PHP, report real hit/forced metrics. This gives truthful hints and metrics, not bandwidth.
Suffix delivery (server + sdk-php, behind a new capability sticky_history_suffix):
- Server: when the sticky worker claims within TTL and the worker's registration advertises sticky_history_suffix, send only events after the last sequence that worker was delivered (recorded at its previous claim), with explicit history_after_sequence and history_last_sequence. Anything else sends full history.
- sdk-php: advertise sticky_history_suffix only with a contiguity check: the suffix's first sequence must equal the cached last sequence + 1, else refetch authoritative history from 0. Old SDKs never get a suffix.
- Recording the delivered boundary needs one column; it would share the unreleased 3.5.0 schema bump with Bind a missing update contract at a run's first accepted workflow task (#962) #1120.
Acceptance: the sdk-php#113 harness on native shows hits and bytes per continuation well below full history, with cold fallback for TTL expiry, another worker, re-registration and a broken prefix.

Owner: server for (1) and the server half of (2); sdk-php for the SDK half of (2). I'll start (1) now; please object here within a day if the suffix design should differ.

sticky_history_suffix v1 wire shape (proposal for sdk-php to build against)

Opt-in (worker → server). Registration lists sticky_history_suffix in capabilities, with capability_manifest.sticky_history_suffix = {"supported": true, "version": 1}. It also needs sticky_execution. Without it the server never sends a suffix.

Server capability. server_capabilities.sticky_history_suffix = {"supported": true, "version": 1} once native ships it.

Poll answer. Two new fields on every workflow task answer to an opted-in worker:

history_after_sequence: integer. 0 means the answer starts at sequence 1 (full history). N > 0 means a suffix: history_events holds only events with sequence > N.
history_last_sequence: integer, the run's last sequence this task delivers through (the last page's last event).

A suffix is sent only when sticky_replay_mode is sticky_hit_expected and the claiming worker was delivered history through exactly N on the run's previous task. N is that previous delivery's history_last_sequence. Otherwise the answer is full history with history_after_sequence: 0. Paging is unchanged: next_history_page_token continues within the suffix, and fetching from token base64("0") always returns full history.

SDK acceptance rule. Use cached + suffix only if all of these hold:

the cache entry (workflow_id, run_id, build_id) exists and starts at WorkflowStarted;
the cached last sequence == history_after_sequence;
the suffix's first sequence == history_after_sequence + 1 and the suffix is contiguous through history_last_sequence.

Otherwise drop the entry, count forced_cold_replay, and refetch from token base64("0").

Vectors

Hit: cached sequences 1..12; answer {"sticky_replay_mode":"sticky_hit_expected","history_after_sequence":12,"history_last_sequence":15,"history_events":[{"sequence":13,...},{"sequence":14,...},{"sequence":15,...}]} → replay 1..15.
Gap: cached 1..11, same answer (after 12) → refetch full.
Overlap: cached 1..13, same answer → refetch full.
Non-contiguous suffix [13,15] → refetch full.
Cold: {"sticky_replay_mode":"forced_cold_replay","history_after_sequence":0,...events from 1} → cold replay, replace the cache.

Server state: the run records the last sequence delivered to its sticky worker. This needs one column, added in the unreleased 3.5.0 schema bump. Server ships PHP parity (#1017 part 1) first, then the suffix behind this capability.

Part 1 (PHP parity) is up as #1140: native records and reports sticky affinity and replay modes exactly as PHP 2.11.8 does, advertises sticky_execution, and reports real metrics. History is still full, as in PHP. Next: part 2, the sticky_history_suffix server half (needs the delivered-boundary column in the 3.5.0 schema bump), built against the wire shape above while sdk-php builds the SDK half. Owner: server.

sdk-php: the consumer for sticky_history_suffix v1 is in durable-workflow/sdk-php#384, built to this wire shape. It advertises the capability and capability_manifest.sticky_history_suffix = {supported: true, version: 1}, and accepts a suffix only if the cache ends at exactly history_after_sequence and the suffix is gap-free through history_last_sequence. Otherwise it refetches from base64("0"). All five vectors are unit-tested. It will merge after the producer lands.

Wire amendment (v1, before anyone ships): the worker reports its cached boundary.

Native's later history pages (/workflow-tasks/{id}/history) are not bounded at the claim. Events appended after a claim (a signal, for example) can reach the worker on a later page. So the server cannot reliably know what a worker cached.

A worker that advertises sticky_history_suffix adds history_last_sequence (integer ≥ 1) to its completion's sticky_cache: the last sequence of the history it replayed and cached for this run. Workers without the capability must not send it. The server refuses it with 422 from a worker that didn't advertise the capability.
The server stores it with that completion (in the completed task's payload; no schema change). On the next claim of the run by the same worker within TTL, it sends a suffix with history_after_sequence = that value. Everything else is unchanged from the shape above, including the SDK's contiguity check, which remains the correctness guard.

Source measurement (2026-11-11, lab4): native Server built from feat/server-1017-sticky-suffix @ 0bacb1d8 (server#1141, SQLite), fresh Server per run. Workflow: 25 sequential 1 s durable timers, so 26 workflow tasks. History bytes = JSON of history_events on every workflow-task answer (polls and completion-carried polls) plus history pages.

SDK	sticky hits	forced cold replays	refetches	history bytes	result
sdk-php#384	25/26	0	0	41,441	completed
sdk-php main (no capability)	0/26	25	0	573,224	completed

That's about 14× less history transferred, and every suffix after the first task was accepted by the contiguity check. This is a source-branch result. #233's acceptance still needs the published pair, and the 1,001-event #113 workload is pending.

Both server parts are on main: #1140 (PHP-parity sticky execution) and #1141 (sticky_history_suffix v1 producer). sdk-php's end-to-end run against the producer: 25/26 hits, 41 KB vs 573 KB history (about 14× less). Remaining: release in native 3.5.0, then sdk-php merges #384 and re-runs the #113 harness on the published pair. Owner: server (release), sdk-php (consumer).

Released in native Server 3.5.0 (https://github.com/durable-workflow/server/releases/tag/3.5.0; index sha256:b5776fea2a4627ebc477b3b350caaf600dc220e7ab75aa29d32b70e404fd7a68 on Docker Hub and GHCR; promoted to 3.5/3/latest after corpus and Compose qualification). Upgrading needs the documented stop/backup/schema-upgrade step.

Remaining for this issue: General merges sdk-php#384 and runs the #113 harness against the published 3.5.0 pair; then close.

Navigation Menu

Repository navigation

Native: no sticky execution, so every workflow task re-sends full history #1017

Problem

Current native behavior (main `d457a214`)

Expected

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Native: no sticky execution, so every workflow task re-sends full history #1017

Description

Problem

Current native behavior (main d457a214)

Expected

Activity

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 7, 2026

rmcdaniel commented on Nov 11, 2026

rmcdaniel commented on Nov 11, 2026

rmcdaniel commented on Nov 11, 2026

rmcdaniel commented on Nov 11, 2026

rmcdaniel commented on Nov 11, 2026

rmcdaniel commented on Nov 11, 2026

rmcdaniel commented on Nov 11, 2026

rmcdaniel commented on Nov 11, 2026

rmcdaniel commented on Nov 11, 2026

rmcdaniel commented on Nov 11, 2026

rmcdaniel commented on Nov 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Current native behavior (main `d457a214`)