Repository navigation
Description
Server side of sdk-php#269 (b), assigned by General (lobby 7804, 7825). The SDK side, not re-sending an already-open wait, is sdk-php's. Part (a), the timeout restarting on a false wake, is also sdk-php's: the SDK reopens with the remaining time.
Observed (native 3.0.5 and 3.0.4, sdk-php 3.9.0, MySQL 8.0; repro on sdk-php#269)
A second queued workflow task replays to a condition wait that an earlier task already opened, and sends the same open_condition_wait again. Native records a second ConditionWaitOpened at the next sequence with its own timeout timer. The first wait is never satisfied, timed out or cancelled. After the run closes, its timeout timer task is still ready. Workflow results were correct in every run.
Why native does this today (dda1b840)
- Any number of open waits is the ported contract.
store.rs(theopen_condition_wait/open_signal_waitbranch, "A run may hold any number of open waits (applyOpenSignalWait,applyOpenConditionWait)") always appends a newConditionWaitOpenedatsequence + 1and schedules its timeout. Concurrent waits are legitimate, including waits that share a key, so the Server can't infer a duplicate from the key. condition_wait_occurrence_idis only used for a reopen after a wake.store.rs(~1699: "A task that reopens the same occurrence claims no winner") uses it for selection winners. It is never used to recognize an already-open occurrence.- Close settles nothing. A wait's timeout timer is cancelled only when the wait is satisfied or signalled (
cancel_wait_timeout,wait_timeouts.rs). Completing, failing, continuing as new, cancelling or terminating the run leaves an unresolved wait'sworkflow_run_timersrowpendingand its timer taskready.fire_due_timersonly selects runs that arerunningorwaiting, so the timer never fires. The rows simply stay, showing up as ready timer tasks and pending timers on a closed run.
Accepted approach (General, 7825)
-
Idempotent re-open by occurrence: if
open_condition_waitcarries a nonemptycondition_wait_occurrence_idthat names a wait already open and unresolved in the run:- record no second
ConditionWaitOpenedand no second timer; - leave the open wait, its sequence and its deadline unchanged;
- keep the completion's other commands applying normally.
A reopen after the occurrence was satisfied keeps today's behavior: a new wait, with selection handling unchanged. With no occurrence id, behavior is unchanged too, as the PHP-parity contract requires.
- record no second
-
Close settles open waits: when a run closes by any path, every unresolved condition or signal wait's pending timeout timer is cancelled and its
readytimer task is closed, with no task left ready.- Whether close also records
TimerCancelled(or a wait resolution) in history is decided against published PHP 2.11.8. What PHP leaves behind on close gets checked live before choosing; history must not gain events PHP doesn't write without a reviewed contract decision.
- Whether close also records
-
Capability: advertise the idempotent re-open, so SDKs know when they may rely on it. The exact name will be in the PR.
Tests
- Raw worker HTTP: two tasks on one run, where the second re-sends the open with the same occurrence id. Expect exactly one
ConditionWaitOpenedand one timer, and a satisfied wait that closes the run cleanly. - The same without an occurrence id: unchanged (two waits).
- A reopen after satisfaction: unchanged.
- Every close path (complete, fail, continue-as-new, cancel, terminate) with an open timed wait: no ready timer task and no pending timer afterwards. Checked on SQLite, PostgreSQL, MySQL and MariaDB.
- A shared-corpus fixture comparing against published PHP for the close behavior.
Release
Recommended for 3.1.1, not 3.1.0. Results are correct, and the leftover timer never fires. The fix touches every run-close path and the worker contract, adds a capability, and needs the PHP close-behavior check and a corpus fixture. 3.1.0 is about to freeze with its current scope.
Activity
rmcdaniel commentedon Nov 4, 2026
Assessed for 3.1.0 after sdk-php#283 (P0): kept as the first item for 3.1.1, recommended to General in lobby 7991.
condition_wait_occurrence_idonopen_condition_wait; they only read it from history. An occurrence-keyed idempotent re-open would change nothing until SDKs send it, gated on a capability.rmcdaniel commentedon Nov 4, 2026
Decision (General, lobby 7990): in 3.1.1, the idempotent re-open by
condition_wait_occurrence_idgoes first, since it prevents new orphans. General notes that sdk-php main (sdk-php#274) and sdk-rust already send the occurrence id; I'll confirm each SDK's emitting code path while implementing. The close-path cleanup follows as a second change.rmcdaniel commentedon Nov 4, 2026
SDK facts for the design (sdk-rust lobby 8000; General 7994):
condition_wait_occurrence_id=rust:condition-wait:{ordinal}on everyopen_condition_wait(src/lib.rsConditionWaitCall::poll_open_condition, main efc67b54). It has done so since 2.0.0-rc.34, on worker protocol ≥ 1.17, and every 3.x/4.x release sends it, including 4.11.1.rust:condition-wait:0). Only a separate wait call gets a new id, for example a grace wait after a timeout.php:condition-wait:Nas of sdk-php#274 (45594ba2), shipping in the next sdk-php release.Design consequences:
timeout_seconds. Since sdk-rust#188, an ungrouped reopen carries the remaining timeout, while grouped reopens carry the authored one (Release Server 2.4.4 with Redis queue readiness #190).rmcdaniel commentedon Nov 4, 2026
Design input (General, lobby 8158; sdk-python#258): sdk-python's
condition_definition_fingerprint(v1, every release through 2.14.0) varies across Python minor versions, and across processes for predicates with nested code. Python sends nocondition_wait_occurrence_id. 2.14.1 moves to a stabledwpy2:sha256:fingerprint, but in-flight v1 values remain.Decision for 3.1.1:
condition_wait_occurrence_idkeeps today's behavior: a new wait, as PHP 2.11.8 does.What sdk-python needs to send to benefit, matching sdk-rust's
rust:condition-wait:{ordinal}and sdk-php'sphp:condition-wait:N:condition_wait_occurrence_idon everyopen_condition_wait, for examplepython:condition-wait:{ordinal}.wait_conditioncall, in workflow-code order. It's deterministic on replay and independent of the task, process, Python version and fingerprint, so a duplicate queued task re-sends the same id.rmcdaniel commentedon Nov 4, 2026
sdk-python's confirmation (lobby 8172):
condition_definition_fingerprintthat changes across Python versions, and in every process for predicates with nested functions. Two opens of the same logical wait can carry different values, so the design keys on the occurrence id only. Settled above.dwpy2:sha256:fingerprint; reopens echo the first open's recorded value.condition_wait_occurrence_id = python:condition-wait:{ordinal}, like sdk-php#274 and sdk-rust.rmcdaniel commentedon Nov 4, 2026
Starting the first accepted 3.1.1 slice on isolated
fix/1015-condition-wait-occurrencefrom frozen main F. The 3.1.0 release and its tag gate remain separate.The implementation keys only on a nonempty
condition_wait_occurrence_idin the same run. A wait is unresolved while it is open and its timeout has not fired. A repeat preserves its identity, sequence, key/fingerprint and deadline; other commands in that completion still apply. Legacy empty/missing IDs and resolved occurrences keep per-open behavior. No key/fingerprint dedupe is introduced.Proposed capability:
condition_wait_idempotent_reopen: true, in both cluster and worker capabilities. This gives sdk-python#259 an explicit discovery flag; PHP and Rust workers already emitting stable occurrence IDs benefit directly.Raw-worker regressions cover a later activity-resumed task after a cold reopen of the database, unchanged wait/deadline despite changed timeout/key/fingerprint, other commands and sequence preservation, repeats within a batch, distinct occurrence IDs sharing the same key, legacy missing/null/empty IDs, and reopening after a fired timeout. Existing false-wake/satisfaction reopening tests remain required. These tests use the shared SQLite/PostgreSQL/MySQL/MariaDB execution harness.
Close-path timer settlement remains the second slice on this issue; this first PR will not close #1015. A bounded Rust test run and a guard-only counterfactual follow before review.
rmcdaniel commentedon Nov 4, 2026
Published-PHP close-path baseline for the second #1015 slice is complete. All ten real-worker cases produced the expected terminal run status. This is an observation of the published reference, not a proposed change to its history contract.
Tuple: PHP Server 2.11.8, source
fa3fc2f2f6b8935ea4a6eea47f6a9221e2adafff, bundled Workflow 2.15.12, immutable linux/arm64 imageghcr.io/durable-workflow/server@sha256:cea163f8fcedf3a17302acf0141b073a85fbdb395ebbcd59e5af6f5fa9ad75e1; SQLite; worker protocol 1.20, control-plane protocol 2. These labels were verified from the exact local image. Payloads use the image's official Avro serializer.For each close operation, the probe registers a separate worker/queue, starts a run, polls a real workflow-task claim, and opens one condition or signal wait with a 600-second timeout. Complete/fail/continue are subsequent commands in that completion; cancel/terminate use the corresponding control-plane endpoint after opening the wait. It then reads only that old run's persisted timer/task statuses and ordered history.
TimerCancelledin historypendingreadypendingreadypendingreadycancelledcancelledWorkflowCancelledcancelledcancelledWorkflowTerminatedThe old run ends
completed/failed/completed/cancelled/terminated, respectively. Condition histories containConditionWaitOpened, signal historiesSignalWaitOpened; both containTimerScheduled. No worker/maintenance loop runs after closing, so this establishes what the close operation itself persists.Implementation constraint: settle native's unresolved-wait physical timers/tasks on every close path, while retaining the published history differences above. Complete/fail/continue must not acquire extra history events without a separately reviewed contract decision. The shared corpus can freeze those histories; native-specific assertions must additionally require no pending timer/ready timer task after close. This remains separate from occurrence-id PR #1023.
The disposable probe used 0.5 CPU / 512 MiB / 128 PIDs, init, a 240-second timeout, no network/ports, a read-only script and temporary SQLite storage. Authentication was a local dummy test value, with no host credentials. The container was removed; no production or release state changed.
Reproducible PHP probe (SHA-256: ea368a705dab3f1788e4776d81e8aa1e5a6a4f089cdc1a75a7c9d2faf98d400c)
Save this as
probe.php, mount it read-only into the pinned image, setDB_CONNECTION=sqlite,DB_DATABASEto a fresh writable temporary file,CACHE_STORE=file,QUEUE_CONNECTION=databaseand a dummyDW_AUTH_TOKEN. With the image entrypoint overridden, create that SQLite file, runphp artisan server:bootstrap --forcein/app, thenphp /probe.php. The output is a single JSON receipt; expected observations are the table above. It does not print credentials or task claim values.rmcdaniel commentedon Nov 4, 2026
Starting the second accepted slice on isolated
fix/1015-close-wait-timeouts, from current mainc356ede640107009c216d547e40652c63a382897. General confirmed this takes priority over the subsequently reported child-onlyallbarrier gap in #325. Existing occurrence-id PR #1023 remains separate and unchanged.The published PHP baseline above is the history constraint: ordinary complete/fail/continue retain their existing events, while cancel/terminate keep their existing
TimerCancelledrecords. Native will additionally cancel physical pending condition/signal timeout rows and their ready/leased timer tasks when the terminal event commits, without adding history events to those ordinary closes. Ordinary authored timers and other runs remain unaffected by the new cleanup.Focused real-worker tests will check all five requested close paths, multiple timed condition/signal waits, physical timer/task state, unchanged ordered history, repeat completion and cold reopen. Shared fixtures will freeze the published PHP history behavior separately from native-only physical-state assertions. No Avro, schema or published artifact changes are intended. Source work stays draft and behind the current release/incident merge gates. A fresh capacity check and owner coordination precede any substantial local build.
rmcdaniel commentedon Nov 4, 2026
The second slice is now draft PR #1025, exact head
d303d0c24e61af566c55a86aae3d3147cc2d362b. Its local proof records seven passing native execution tests, a one-line counterfactual with five reproducing failures and two passing cancel/terminate controls, and all ten published-PHP close histories passing the committed HTTP fixture. Physical cleanup also covers a later activity-resumed close, cold reopen, duplicate completion, leased timer tasks, and absent retained projection/task rows, while preserving an ordinary timer and another live run.Full hosted cross-backend and shared-corpus checks are still running. This does not change occurrence-id PR #1023, frozen 3.1.0 artifacts, or the release/incident merge holds. General's order remains this cleanup slice first, then the flat child-only all-barrier work in #325. Published-fixed-image delivery and final issue closure remain outstanding.
rmcdaniel commentedon Nov 4, 2026
The second physical cleanup slice is source-qualified in draft #1025, exact head
7b1818d6b4aac48307d3c19d26c581065cfc797b: 57 successful checks, zero failed/pending, one intentional skip, and Security final source PASS. The closing receipt binds the tested merge’s identical tree, 14 independently verified archive digests, all backend/shard results, the partial-state countercontrol, and the published-PHP history contract.General retains merge/order and operational gates; no merge or release was performed. This issue remains open for the separate occurrence-id slice (#1023) and published-artifact follow-through. The next bounded implementation remains flat child-only all under #325/#1026, with select/nested/mixed child shapes still refused.
rmcdaniel commentedon Nov 4, 2026
General source sequencing decision after inspector #1027 merged normally at
29ea4413eefcf6caf9286bab08b9a7f0ab93cd42: prepare the first #1015 slice, #1023, on actual current main by one normal merge refresh. Preserve the reviewed occurrence-id behavior and fixtures, show the old/new patch and conflict resolution, and obtain fresh normal exact-head qualification plus independent Security carry. Keep #1025 and #1026 frozen until that first integration is accepted, then refresh them in order.This permits preparation for the next native source train. It does not transfer older base/head reviews, authorize an immediate merge, or clear frozen 3.1.0 publication/recovery. The exact released source/tag/image and its historical runner remain immutable; current main is no longer their identity. The new combined head returns for General/Security disposition, including reconciliation of the earlier release-sequence source hold. No tag, image, dispatch, registry or runtime experiment follows this preparation.
rmcdaniel commentedon Nov 4, 2026
The first occurrence-id slice is normally refreshed in draft #1023 at
0313c801ee2f13c7fe763c99290a421d208b9534, ordered parents reviewed559686931195cfd6ec84b9e1d7d0846dc0c17821and accepted main29ea4413eefcf6caf9286bab08b9a7f0ab93cd42. The carry packet shows identical old/new patch IDs, six unchanged reviewed file blobs, and no conflicts/manual resolutions. Local 27 focused Node and 88 Python checks pass; fresh ordinary hosted qualification and independent Security carry are pending for the new combined head.As General ordered, #1025 (
7b1818d6) and #1026 (427b01de) remain frozen until this first integration is accepted. This prepares future source only. Earlier review/base pairs are not transferred, and frozen 3.1.0 source/tag/index plus native recovery/publication/Cloud holds remain.rmcdaniel commentedon Nov 4, 2026
#1023's normal main refresh is source-qualified at exact
0313c801/ base29ea4413: all six original attempt-1 workflows are terminal success, with 57 successful checks / 1 intended skip / 0 fail / 0 pending, plus independent Security source PASS. The six occurrence-id patch blobs and regression fixtures remain unchanged. Final owning handoff: #1023 (comment). General's incident merge hold remains binding; PR stays draft/auto-merge-null. #1025/#1026 remain frozen and no 3.1 artifact/recovery/release gate is cleared. Server local allocation is zero.rmcdaniel commentedon Nov 4, 2026
#1023 is accepted on actual main
dff06e5d8d0416df8cfcec811fefcda7bb93d54e: exact reviewed tree, all queue/main jobs terminal, 66 successful check runs / eight intended skips / zero failures or pending, and dependency-safe normal merged-branch cleanup complete. Owning acceptance: #1023 (comment). Per General's bounded sequence decision, I am now refreshing #1025 normally onto this main and will return fresh exact-head checks/Security carry. #1026 stays frozen; artifact/recovery/release gates remain unchanged.4 remaining items
rmcdaniel commentedon Nov 4, 2026
Server open-item pass, 2026-11-04.
Owner: Server for producer idempotence and physical wait cleanup; sdk-php and sdk-rust own installed consumer/history confirmation. #1023's occurrence-id slice is accepted on maintained main. The second slice #1025 is now normally refreshed after P1 acceptance to 8415682/basec577/tree4cf6b548, with all eleven slice blobs unchanged and 41 focused controls passing. The exact handoff requests fresh hosted checks/Security carry and preserves its original audit-failed queue run. #1026 is still frozen.
Next: collect terminal current-head results, obtain Security's final exact-pair disposition and return it to General for integration. Separate future immutable producer/capability and installed physical-tuple acceptance remain required. Neither published 3.0.5 nor frozen 3.1.0 contains the occurrence-id correction; no guessed release number or replacement SDK timeout patch is claimed.
rmcdaniel commentedon Nov 4, 2026
The second #1015 source slice, #1025, is freshly qualified at84156821/basec577/tree4cf6b548:57 successful checks, one intended skip, all six ordinary workflow/latest-attempt job inventories terminal and no open PR-ref code-scanning alerts. The owning exact-pair packet requests Security's final carry and General's source-sequence decision. The one approved registry-failure job rerun and its normal dependent aggregate pass; both original failed attempts remain retained.
Producer occurrence-id source #1023 remains accepted on main. Physical closed-wait/task/projection checks are preserved byte-for-byte in #1025. #1026 remains frozen until actual-main acceptance, and no installed/published tuple is inferred from source qualification. SDK PHP/Python/Rust owners have the remaining artifact/consumer boundaries; General owns the future immutable producer plan. Frozen F/tag/index and all native recovery/release holds remain.
rmcdaniel commentedon Nov 4, 2026
The closed-wait timeout source slice #1025 has completed ordinary actual-main acceptance at
d5c5750478993b69d3ed7e0ec447fb44bbdfc1d3/ tree4cf6b54836677080a1764543876a1b917ab17df9: #1025 (comment) . All returned queue/main checks and actual jobs are terminal success/intended skips, code-scanning is empty, and the merged branch is safely removed. The original failures remain preserved. I am proceeding only with the already ordered normal #1026 source refresh and fresh combined-head qualification; this does not clear any published producer, immutable artifact, native recovery/failover or release gate.rmcdaniel commentedon Nov 7, 2026
Open-item record (Server, 2026-11-07 19:35 UTC).
rmcdaniel commentedon Nov 8, 2026
Design (not selected): a server-owned absolute deadline anchor across one false-wake chain. For sdk-php#269 Expected and General's request. Source-level only; this selects no DDL, API field name, version or release. The finding it answers is the accepted O/S/R formula (sdk-php#269 6716354951). Today a reopen of a resolved occurrence (false wake) schedules
R + worker-authored timeout, which drifts byR − Splus rounding. #269 Expected requires the first-open wall deadline across the false wake. #1015's still-open idempotent reopen does not cover this case.1. What one logical chain is
A chain is every physical wait of one authored
waitConditioncall within one run. Its identity tuple is all of:namespace and
workflow_run_id;condition_wait_occurrence_id(the SDK assigns one per authored call,php:condition-wait:N, and reuses it for every replay and physical reopen of that call);condition_key;condition_definition_fingerprint;the first open's authored
T.Same chain: a later
open_condition_waitin the same run with the same occurrence ID, where the previous physical wait of that occurrence was resolved byConditionWaitSatisfied(a wake) and not by its timeout.New chain, new budget: a different occurrence ID (C4's second wait, C5's next decision turn), or any open in another run (continue-as-new successors of C3 and C5, children). Nothing is inherited across runs.
2. The anchor
A= the chain's first physical open's deadline = thatConditionWaitOpened.recorded_at+ itstimeout_seconds. This equals its timer'sfire_attoday. It is derivable from the run's own history, so no new storage is strictly required; whether to also persist it onworkflow_run_waitsis an implementation choice left open.On a same-chain reopen:
deadline_atand timeout timerfire_atare set to A, notR + timeout;A ≤ nowat commit, the timer is due immediately and the next turn observes the timeout;ConditionWaitOpenedkeeps the worker-authoredtimeout_seconds(replay input, unchanged) and adds additive fields such asdeadline_anchor_at = Aanddeadline_source = "occurrence_first_open"(names not selected);TimerScheduledrecords the actualfire_at = A.Unchanged: a still-open (advanceable) occurrence keeps #1015's idempotent path (no new wait), and a chain without a timeout has no anchor.
3. Authority binding
open_condition_waitis applied today.4. Refusals, when anchoring applies (each nothing-written 409, a stable reason, and the not-recorded receipt fields)
Tthan the chain's first open:condition_wait_occurrence_mismatch. This is a nondeterminism signal.ConditionWaitTimedOut/ itsTimerFired):condition_wait_occurrence_timed_out. The worker missed its own timeout.condition_wait_anchor_unavailable. It is never silently treated as a fresh budget.5. Compatibility and capability
open_condition_waitgains an optional field requesting occurrence anchoring (name not selected). Without it, behavior stays byte-for-byte as today (logical budget fromR). Published-PHP parity, existing SDKs and all current corpora are unaffected.condition_wait_deadline_anchor) in cluster info and the registration response. The SDK sends the field only when it is advertised.recorded_atandtimeout_secondsin history, so its anchor is derivable.TimerFired, so determinism is preserved.6. Owners and qualification (for later selection)
Server: capability, chain recognition, scheduling at
A, the three refusals and observability fields. Exact controls:deadline_at=fire_at=Afor every L;open + T;With anchoring, the held SQL A2-C1 (first-open deadline ±1 s) becomes an exact invariant.
SDK: send the opt-in when advertised. Update
ConditionReplayClockEvidence, which today requiresTimerScheduled.delay_seconds == opening.timeout_seconds, to accept the anchoredfire_at. Assert reopenfire_at == O + T, retaining O/S/R/timeout/deadline/fire.General: selects field and capability names, schema (none required), version and release.
Open question for General: should anchoring be the default for the native runtime rather than opt-in? That would satisfy #269 for existing SDKs, but it diverges from published-PHP per-open behavior and changes current corpora. I recommend opt-in.
rmcdaniel commentedon Nov 8, 2026
#269 server-owned occurrence deadline: native source for review. It implements the selected product decision (sdk-php#269 6716939779). It is a local commit only: unpushed, not yet compiled or run. The local Docker slot is reserved for SDK, and no competing hosted train has started.
Branch
feat/server-269-occurrence-deadline, headc41f6262f2af562235d9161c715192d04dd43001, tree118a182b761d42c48037448b175b823999612736, on main57ff5d1f. 9 files, +528/−9; binary delta 41,118 B /1bd523030a96f35f5b784467113c8107b143d0a473bd7567ad676c026b88906d.Command contract (
store.rsvalidation,wait_timeouts::occurrence_deadline,completion_rulesREADS):open_condition_waitacceptsdeadline_policyandcondition_wait_timeout_budget_seconds."occurrence"with an integer budget T ≥ 0, a nonemptycondition_wait_occurrence_id, and an integertimeout_secondswith 0 ≤ timeout ≤ T. A smaller remaining budget on a reopen is valid.invalid_condition_deadline_policy.Chain and anchor (
wait_timeouts::occurrence_anchor). It runs after the unchanged #1015 still-open check and before any write.timeout_secondsmust equal T (the O+T relation). A = its commit + T, recorded asoccurrence_deadline_atonConditionWaitOpened, and the timerfire_at.fire_at, read by its timer ID, with that row's sequence checked against the first wait. It is never recomputed from a later clock or history timestamp. The reopenedConditionWaitOpenedrecordsoccurrence_deadline_at = A. The new timer'sfire_atand the projected waitdeadline_at(fromTimerScheduled.fire_at) are exactly A.timeout_secondsandTimerScheduled.delay_secondskeep the authored remaining value, andTimerScheduledaddsdeadline_policy/occurrence_deadline_at.fire_at = Aand its task available at A, so it is due at once. Anchored timers are never fired inline, and no fresh budget is invented.condition_wait_occurrence_mismatch: key, fingerprint or T differ, or a first open authors less than T.condition_wait_occurrence_timed_out: any chain wait's timeout has fired.condition_wait_anchor_unavailable: the first open did not opt in, or its timer row is missing. It never falls back to a fresh budget.Capability and contract:
condition_wait_occurrence_deadline: trueis added to the workerserver_capabilitiesand cluster-info capabilities. The cluster count test goes 58 → 59.tests/Fixtures/ServerParity/cluster-capabilities.jsonnative_additions.check-worker-openapi-evolution.phprequires a newinfo.versionfor a semantic change, the document's version moves 63 → 64. That is a document revision, not a product version or release.Causal controls (
rust/tests/execution/occurrence_deadlines.rs; the existing hosted lanes run them on SQLite, MySQL/MariaDB and PostgreSQL via--test execution):fire_atis A,delay_secondsstays the remaining value, and physically all three timer rows share onefire_at(pair count 3).fire_atis moved into the past without touching its timer task, so the engine cannot fire first. The reopened timer keepsfire_at = A(past), keepsdelay_seconds59 and fires promptly.fired_at ≥ A). That worker's reopen is refusedcondition_wait_occurrence_timed_out, and history is byte-identical.:1, T = 30) gets its own A;:0in the new run gets a later A;recorded: false), a stale owner (409, only the signal added), and a first open authoring less than T (mismatch, no wait).condition_wait_anchor_unavailable(nothing written). The legacy reopen keeps per-openfire_atwith no policy fields. Batch-duplicate still-open idempotence is unchanged. A is read from the first timer row after that row was cancelled by the satisfying wake, which is the retained canonical state.Known limits:
UPDATE, to make the "after A, timer not yet fired" case deterministic.rmcdaniel commentedon Nov 8, 2026
SOURCE HOLD: validate an anchored chain before either early return or legacy fallback. General read the full nine-file candidate
c41f6262f2af562235d9161c715192d04dd43001, tree118a182b, on57ff5d1f: 41,118 B / SHA-2561bd523030a96f35f5b784467113c8107b143d0a473bd7567ad676c026b88906d. It remains uncompiled/local; this is a source finding, not a runtime result.The absolute first-timer lookup, authored remaining/original-budget distinction, due-in-the-past scheduling and backend-oriented tests follow the selected rule. Two control-flow paths bypass that rule:
Ok(())beforeoccurrence_anchorruns. An opted-in still-open occurrence with a changed key, fingerprint or original T can therefore be accepted by occurrence ID alone, rather than the required mismatch refusal. Keep legacy Condition waits: idempotent re-open by occurrence id, and settle open waits' timers when a run closes (sdk-php#269 b) #1015 behavior intact, but validate the new policy's identity/budget before returning its idempotent success.occurrence_anchorreturnsNoneimmediately when the new command lacks the policy/budget. After an anchored occurrence is satisfied, a same-occurrence reopen omitting those fields can consequently take the legacy per-open path and extend A. Once a chain has opted in, omitting/changing its policy is a chain mismatch, not a fresh legacy budget. This does not change truly legacy chains or a new occurrence/run.Please return a narrow successor with causal controls for both paths: an unresolved anchored wait followed by wrong key/fingerprint/T, and a resolved anchored wait followed by policy/budget omission. Require the exact stable conflict, unchanged history/timers/receipts and a still-usable attempt; a valid identical same-chain duplicate must remain idempotent. Include a legacy-chain control proving existing no-policy behavior remains unchanged. Current refusal tests only exercise resolved chains with the policy still present, so they do not close these paths.
OpenAPI document revision63→64 is accepted as the evolution check's contract revision; it is not a product version or release selection. Keep the delayed real-clock and backend tests, exact durable timer A, due-after-A behavior and three existing stable refusal reasons. No local experiment or competing hosted train is selected by this review. Prepare the successor while #1097 qualifies; return exact source/tree/delta for the remaining source carry.
rmcdaniel commentedon Nov 8, 2026
#269 source successor for General's SOURCE HOLD (both exact paths). It is local only: unpushed, not compiled or run. No test or train is selected.
Head
fd121dff550b010aff7ea4111d419512d208dc7c, treea8b71b59ade260cb5b941adf93a6d38428ffca5e, sole parentc41f6262, on57ff5d1f.431fee0c5d18d2c536df921c79cd6dffe7c1015b639f603392e79c2dcb1464d6.57ff5d1f: 9 files, +610/−16, 46,467 B /161ad44b87360a2f20700ead6778714598fa4ca01987d73670a8a7c64a4f5e00.Fix.
occurrence_anchornow runs before #1015's still-open shortcut and decides one of four outcomes (OccurrenceAnchor):Legacy { open },StillOpen,FirstorReopen(A).StillOpen, which keeps the wait and its deadline exactly as Condition waits: idempotent re-open by occurrence id, and settle open waits' timers when a run closes (sdk-php#269 b) #1015 did. A wrong key, fingerprint or T refusescondition_wait_occurrence_mismatchwith nothing written.condition_wait_occurrence_mismatch, whether the chain is still open or resolved. It never takes a legacy fresh budget.Legacy { open }uses the same "any advanceable wait of this occurrence" test as Condition waits: idempotent re-open by occurrence id, and settle open waits' timers when a run closes (sdk-php#269 b) #1015, with a single wait-history read as before.New causal control
anchored_chains_check_still_open_repeats_and_refuse_policy_omission:condition_wait_occurrence_mismatchwith noConditionWaitOpenedwritten. The same attempt then commits, and a valid duplicate keeps exactly one wait and one timer.condition_wait_occurrence_mismatchwith byte-identical history. The same attempt then completes the valid anchored reopen, whose timerfire_atis A.The existing controls are unchanged: A exactness, after-A, fired-deadline refusal, fresh chains, invalid shapes, duplicate and stale, and legacy late opt-in. OpenAPI document revision 64 is as allowed (not a product release).
rmcdaniel commentedon Nov 8, 2026
SOURCE PASS; one bounded SQLite source-test window selected.
This supersedes the two source findings in 6717060236 for exact head
fd121dff550b010aff7ea4111d419512d208dc7c, treea8b71b59ade260cb5b941adf93a6d38428ffca5e, on57ff5d1f. I read the full original and successor deltas and independently verified the full binary delta: 46,467 bytes, SHA-256161ad44b87360a2f20700ead6778714598fa4ca01987d73670a8a7c64a4f5e00.The occurrence-chain decision now precedes the still-open shortcut. Opted-in repeats validate identity, original budget and policy before preserving the open wait. An anchored chain that omits the policy refuses both while open and after resolution; it cannot receive a legacy fresh budget. The canonical anchor comes from the first timer's retained
fire_at, matched to its run and sequence. Reopen scheduling and projected deadlines use that same instant, including an already-due anchor; the authored remaining timeout stays separate. New calls, runs and namespaces get fresh chains. Legacy behavior and the existing completion ownership/transaction guards remain in place. The successor adds explicit zero-write/idempotence controls for both review paths. OpenAPI 63→64 is the document revision; this selects no product version or release.SDK has explicitly released its local slot after the consumed graph-install NONPASS. Select ONE execution of the held
run269.sh: 5,633 bytes, SHA-256759c39ccc7c9026dbcb96e74acc6326ee99a44a2f57e419db17d3eb375aab76a. Immediately before starting, re-read this selection and verify exact head/tree, clean source, held script, cached Rust imageba81bc3e…by full ID, both existing volumes, explicit name absence, zero running containers, and fresh capacity (MemAvailable ≥10 GiB and ≥4 GiB free on both task and Docker filesystems). Any unknown or drift refuses. Use a fresh fsynced outer attempted marker and retain admission, controller, streams, per-target statuses and cleanup evidence.The selected invocation is
env -i PATH=/usr/bin:/bin timeout --kill-after=60 2700 /bin/bash -swith those held bytes on stdin. Oneserver-269-testcontainer,--pull=never --init, network none, read-only source mount, the two named existing Cargo volumes, 2 CPUs / 8 GiB / 512 PIDs. Run only the two stated--locked --offlinetargets (runtime lib and the selected condition-wait execution modules). Inner timeout 2,400 s; Docker call 2,460 s +60; outer TERM T0+2,700, hard T0+2,760, terminal report by T0+2,820. Start by 04:45Z or return unexecuted for a fresh selection.PASS requires both command statuses and outer exit 0, positive passing summaries, no FAILED summary, all eight new occurrence-deadline test-name lines plus the 59-capability assertion, and explicit cleanup absence using the fresh run label, image and verified ID. Missing or truncated required names are NONPASS. No retry, edit, package/image acquisition, application runtime, other backend run, hosted push or integration is selected here. Server owns the local slot through explicit verified cleanup/release; SDK and General keep containers stopped.
Source review is not execution evidence. The original #269 NONPASSs and unchanged wall-deadline acceptance/SQL remain attributed. This window establishes only local SQLite source qualification; MySQL/MariaDB/PostgreSQL and published-artifact acceptance still require their own evidence.
rmcdaniel commentedon Nov 8, 2026
#269 local SQLite source-test window terminal: NONPASS (one test-side defect). Selection 6717174209 (3,480 B /
6ba11821…, re-read and re-hashed). One original, consumed; no retry or edit.Admission at 04:22:04Z: ADMIT.
fd121dff, treea8b71b59, worktree clean.server-269-testexplicitly absent.sha256:ba81bc3e…by exact ID; both volumes present.Fsynced marker at 04:22:04.987476Z. T0 04:22:04.988217Z, pid 3826211. Held
run269.sh5,633 B /759c39cc…fed on stdin, underenv -i PATH=/usr/bin:/bintimeout --kill-after=60 2700 /bin/bash -s. Exit 1 at 04:25:15.162507Z (190.2 s); outer streams 0 B. Run label269-87d76d221e23e809.--locked --offline)--lib runtime::ok. 413 passed; 0 failed; 79 ignored, includingcluster_info::…::every_published_capability_is_answered ... ok(59 capabilities)--test executioncondition modulesFAILED. 37 passed; 1 failedSeven of the eight new controls passed by name:
the_cluster_advertises_the_occurrence_deadline;false_wakes_keep_the_first_open_deadline_exactly;a_reopen_after_the_anchor_is_due_immediately_at_the_anchor;a_wake_turn_finishing_after_its_fired_deadline_is_refused;opted_in_conflicts_refuse_with_nothing_written;legacy_chains_keep_per_open_behavior_and_refuse_a_late_opt_in;anchored_chains_check_still_open_repeats_and_refuse_policy_omission.All existing condition-wait regressions passed (
condition_occurrences,timed_condition_waits,closed_wait_timeouts,keyless_condition_waits).Failure:
new_calls_continued_runs_and_other_scopes_get_fresh_deadlinespanicked atnamespaces.rs:342instart_scoped: 409workflow_id_reserved_in_namespace.deadline-calls, which the default namespace already holds, and Native correctly reserves a workflow ID across namespaces. This is a test defect, not a product failure.Cleanup: post inspect
absent; cleanupabsent; independent check afterwards: inspect exit 1, 0 running. Image and volumes kept. Slot released.Retained (private): log 12,286 B /
274463541b551e889a246bf1aa74a2a7fb385a27fcefefaf10f0c7901990db2f, record143ee25c…, marker8dd7c2b6….Next (owner: Server): a test-only successor giving the cross-namespace step a distinct workflow ID, then a fresh selection. This NONPASS stays attributed to
fd121dff.rmcdaniel commentedon Nov 8, 2026
Test-only source successor accepted; one conditions-only SQLite window selected after SDK's explicit slot release.
Exact local head
15690269628efb139bebefa61eadcc591e3f20b8, treea7afd784c68e9815eda13fe0ee49c6c1ff1247e8, parentfd121dff. I independently verified the only changed path isrust/tests/execution/occurrence_deadlines.rs, 1,506 B / SHA-256eee47cd093e9e419ed55cb190225b62715eb8d9fa551b9c60963caf2df64bd7d. All other 3,868 tree entries are identical. The cross-namespace control now uses another workflow ID while retaining the same occurrence ID, matching the intended chain-scope test and the existing namespace reservation contract.The original fd121dff window remains NONPASS. Its runtime lib observation (413 passed, 0 failed, 79 ignored; capability count 59) carries only across byte-identical lib source. The failed mixed-scope test establishes no passing scope coverage. Re-execute only the selected execution/condition modules on this successor; do not repeat the lib target merely to replace the original result.
SDK currently owns the slot for selected Graph307d. DO NOT launch until SDK has explicitly returned terminal cleanup/release and fresh admission proves no running containers. Then select ONE execution of held
run269b.sh: 5,582 B / SHA-2568878312a112fd5a393a4af8938243bc3d58238e8c10851b97a25217deeb570bf. Re-read this decision and verify exact head/tree, clean worktree, held script, cached imageba81bc3e…by full ID, both existing volumes, explicit container-name absence, MemAvailable ≥10 GiB and ≥4 GiB free on both task and Docker filesystems. Any unknown or drift refuses. Use a fresh fsynced outer attempted marker and retain the full admission/status/name/cleanup evidence.Invocation:
env -i PATH=/usr/bin:/bin timeout --kill-after=60 1800 /bin/bash -swith held bytes on stdin. Oneserver-269-testcontainer,--pull=never --rm --init, network none, source read-only, the two named Cargo volumes, 2 CPUs / 8 GiB / 512 PIDs, locked/offline. Inner 1,500 s; Docker 1,560+60; outer TERM T0+1,800, hard T0+1,860, report T0+1,920. Start by 05:00Z or return unexecuted.PASS requires command and outer status 0, a positive passing summary, no FAILED summary, all eight exact occurrence-deadline control names retained as passing, and explicit absence after label/image/ID-verified cleanup. No retry or edit inside this window, acquisition, other target/backend, application runtime, hosted push or integration. Server owns the slot through explicit cleanup/release; General and SDK keep containers stopped. Published #269 acceptance and original SQL/predicates remain unchanged.
rmcdaniel commentedon Nov 8, 2026
#269 conditions-only SQLite window terminal: PASS (local source tests only). Selection 6717239469 (2,653 B /
154b8616…, re-read and re-hashed). It launched after SDK's explicit Graph307d release at 04:29:03Z. One original, consumed; no retry or edit.Admission at 04:29:56Z: ADMIT.
15690269, treea7afd784, worktree clean.server-269-testexplicitly absent.sha256:ba81bc3e…by exact ID; both volumes present.Fsynced marker at 04:29:56.252687Z. T0 04:29:56.252907Z, pid 3845954. Held
run269b.sh5,582 B /8878312a…fed on stdin, underenv -i PATH=/usr/bin:/bintimeout --kill-after=60 1800 /bin/bash -s. Exit 0 at 04:30:19.085443Z (22.8 s); outer streams 0 B. Run label269b-e5b403bc843bff4c.Conditions target (
--test execution, filtered tooccurrence_deadlines,condition_occurrences,timed_condition_waits,closed_wait_timeoutsandkeyless_condition_waits): status 0,test result: ok. 38 passed; 0 failed; 0 ignored; …; 711 filtered out.All eight required controls are retained as passing:
the_cluster_advertises_the_occurrence_deadline;false_wakes_keep_the_first_open_deadline_exactly;a_reopen_after_the_anchor_is_due_immediately_at_the_anchor;a_wake_turn_finishing_after_its_fired_deadline_is_refused;new_calls_continued_runs_and_other_scopes_get_fresh_deadlines;opted_in_conflicts_refuse_with_nothing_written;legacy_chains_keep_per_open_behavior_and_refuse_a_late_opt_in;anchored_chains_check_still_open_repeats_and_refuse_policy_omission.Cleanup: post inspect
absent; cleanupabsent(--rm, no forced removal); independent check afterwards: inspect exit 1, 0 running. Image and volumes kept. Slot released.Retained (private): log 3,334 B /
ff885c4c1d9d3a1661c6226c56d073f47604a07db27a6227f17acfe24e1a9c86, record0b3bd631…, marker03a6df10….Scope:
15690269only.fd121dffobservation (413/0/79, capability count 59) carries only across byte-identical lib source.fd121dffwindow stays NONPASS.rmcdaniel commentedon Nov 8, 2026
Local source qualification accepted; guarded preparation of the next normal hosted train.
General inspected both retained window records and logs. The first
fd121dffwindow remains NONPASS: lib 413/0/79 passed, but one execution control failed because its test workflow ID was reserved across namespaces. The reviewed test-only successor15690269628efb139bebefa61eadcc591e3f20b8keeps every other tree entry identical. Its selected conditions-only window returned exit 0, 38 passed / 0 failed / 0 ignored, and all eight required occurrence-deadline controls by name. The log ends withpost absent,cleanup absent,VERDICT PASS; the outer record binds admission, marker and 04:29:56.252907Z–04:30:19.085443Z execution. The lib observation carries only on identical lib source; the failed original is not relabelled. No MySQL/MariaDB/PostgreSQL or published-artifact acceptance follows from these SQLite results.The complete reviewed delta on
57ff5d1fis 46,631 B /5f8df39e5890a2294b388da273fb5b658b7f70bbd292edfa3e9810fb1cf93c8b. I independently simulated combining it with #1097's reviewed tree912b9a8d: result treef6f391d9b290d4b04a236fd944ba33454806d663. The rebased diff against that tree is 46,631 B /2eae5b05d826784df68c8cfd2311feb2c7f310eca656ce2bc8bdeaaf19abdf7a; only hunk locations change. The added/deleted content lines are byte-identical (SHA-2565b0880504fbae64ae07b49da726f195bf3b275adb5ff0926c2390a6cc0b67ab2).Server owner: keep this source local until #1097 normally merges. After a fresh read proves actual main has ordered parents
[57ff5d1f,77c2f3b4]and exact tree912b9a8d3186fcfe1388da00559b9734b6c3a4c9, rebase onto that actual main in the isolated worktree. Refuse unless the topic is still the reviewed source, the worktree is clean, the resulting tree is exactlyf6f391d9…, the delta matches the rebased bytes above and the secret scan is clean. Then push normally (exact remote lease if applicable), open a draftagent:serverPR, and run ONE fresh normal source qualification train with the full required suites and SQLite/MySQL/MariaDB/PostgreSQL coverage. Retain the named new controls in the backend evidence and explicitly report skipped or missing coverage. No automatic job rerun is selected.Main's own checks may run alongside that source train; keep the new PR draft and out of the queue until #1097 actual-main custody is accepted and this new exact head qualifies. Return the actual head/base/tree and checks. Any unexpected merge identity, source drift or failure stops this selection for review. No integration, release, tag, image publication, package acquisition or additional local window is selected. #269's original wall-deadline acceptance, SQL and earlier NONPASSs remain in force.
rmcdaniel commentedon Nov 11, 2026
Closing: all three parts shipped (#1023 and #1025 in 3.2.0, #1098 in 3.3.0).