Skip to content

Add a consumed scalar business replay clock - #289

Merged
rmcdaniel merged 7 commits into
mainfrom
feat/215-business-replay-clock
Nov 6, 2026
Merged

rmcdaniel merged 7 commits into
mainfrom
feat/215-business-replay-clock

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Workflow code can now deliberately use ctx.replay_now() for business deadlines and elapsed time at supported consumed boundaries. The additive accessor returns the exact UTC-aware activity/timer instant or explicit redrive start and raises exported ReplayClockUnavailable when authority is missing, ambiguous, unsupported or outside the matching execution scope. Legacy ctx.now(), UUIDs, commands, history and cancellation clocks keep their existing behavior.

The clock indexes identities, then reads time only on the actual scalar result/failure branch. Equal/backward times retain their precision. Redrive supports retained start-only history and the finite same-identity StartAccepted/WorkflowStarted prefix shared by Core and native. The prefix does not identify the producer. Correlated reused activity success and timer pairs retain original time; failures use their own event time. Worker recorded_at and public-history timestamp forms are supported. Query callbacks require the supported final inspected prefix. Scoped restoration and task/thread identity prevent another replay or callback from borrowing an instant. Ordinary entry still lacks a proved canonical run.started_at carrier; local/inline/grouped and other families remain explicitly unavailable.

Validation: all 21 current-head checks pass, including Python 3.10–3.15, integration and teardown, portal preview, corpus, CodeQL and Sonar. Direct Sonar issue inventory is zero. Python 3.11 passes 5635 tests with two optional skips; full local units pass 5634 with three optional-dependency skips and 130 integration tests deselected. Ruff/mypy pass. The 51 clock controls include cyclic-discard cleanup, original-error preservation, generator ContextVar tokens, identity/prefix refusals, complete Core timer reuse and both timestamp representations. Disposal uses a synchronous thread-local clock guard so finalizers do not write ContextVars; generator tokens and the existing cancellation replay context keep their behavior.

The identical official consumer runs 103 tests on four product sources: base 6f30853b has three behavioral failures; pre-prefix-fix 089d7a9 has two; wrong-inference 199caf6 has exactly one on the complete Core timer case; candidate passes all 103. No collection/import failures supply a negative. Existing corpus selectors/guards and Core's successor checker/controls remain unchanged. Native's named evidence profile stays activity-only. A finite Core prefix adapter validates its acceptance row before applying the unchanged successor checker; it rewrites no payload/time. Every redrive fixture remains registered and checked. Replay inventory grows 104→107; codec stays 43.

Refs durable-workflow/.github#215. Follows the source cohort, confirmed Python API and existing consumed-event/callback decisions. Full published cross-SDK conformance, later families and guides remain open. Stable docs stay bound to published SDK 2.14.1. No new wire field, marker, package/release, driver/TLS execution or #184 route is selected.

@rmcdaniel

rmcdaniel commented Nov 6, 2026 •

Copy link
Copy Markdown
Member Author

The original 819b96b head remains NONPASSING. Python3.11 job127682115579 reported 130 failures: the new scope's _ACTIVE.reset(token) raised LookupError during unwind and masked existing replay errors. The other source/control results do not override that failure. This return establishes the failing reset path; it does not claim a proved interpreter root cause.

Successor 9a6ab39 restores the saved prior frame value directly, without deleting an absent binding or catching a cleanup error. A new cyclic-discard control checks generator cleanup, parent-clock restoration and preservation of the original NonDeterministicReplayError. All 223 focused clock/corpus/wait controls pass locally after the change, plus Ruff/mypy. The full unit suite and a fresh normal supported-version matrix are running; actual Python3.11 success is required before acceptance. The first failed run is retained, with no unchanged-head retry.

The exception-test Sonar finding is corrected in this successor. Separating scalar row validation reduced the remaining lifecycle function to complexity20, still above15; a further lifecycle extraction follows on the same PR. No rule suppression, skip, corpus-policy or validator weakening was introduced. The same fixture/consumer and existing legacy clocks/errors remain enforced. No main/publisher/operating or #184 change is involved.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Bounded producer-field review of 819b96b2c6ae5e2363791336933322a63349308c, carried across the inspected clock-only delta to current 089d7a9022f7106208604483f929cc5a9ecefc2d. This is source inspection of clock field/time assumptions, not an execution or operating review.

One concrete native redrive mismatch remains in BusinessReplayClock.__init__: the index == 0 requirement rejects the native successor history. At accepted Server 3d4d4afaf2b24aaa5443eac154e253b011924738, redrive.rs appends StartAccepted before WorkflowStarted, followed by reused completions. store.rs returns those stored events in sequence without filtering that prefix; Python's inspected replay path also keeps the input list. The current clock therefore leaves redrive entry and reused-completion time unavailable for this actual producer shape. Please add the bounded StartAccepted + WorkflowStarted shape with matching original workflow/run/command identities and contiguous history sequences, retaining refusal of unrelated prefixes, conflicting identities and duplicate starts. Do not simply skip arbitrary leading rows.

The remaining requested mapping is coherent within its declared subset:

  • Native ordinary activity/timer events use positive payload.sequence with activity_execution_id / timer_id. Ordinary timers omit timer_kind; condition/signal timeout kinds remain distinct.
  • Native redrive writes the explicit workflow/run/command/link/source/resume fields already checked here. Reused ActivityCompleted carries a new execution ID, original reused_activity_execution_id, immediate reused_from_run_id, and the retained original reused_recorded_at. Current native redrive supports the activity-only prefix; this is not a native reused-timer claim.
  • Native worker history uses top-level recorded_at in UTC with microseconds. Public history uses top-level timestamp for that same stored value. Scalar timestamp-first/recorded-at fallback matches these separate representations. The redrive start's insertion-time read currently accepts only recorded_at; public-history redrive input must either receive equivalent explicit support or remain documented unavailable.
  • Completed reused steps use their retained original time; an ordinary failure/timeout/cancellation uses its own event time. Core's reused-timer mapping is separate: accepted Core 98bd98688663c005cb048d88f18119c35b6973b1 FailedRunRedrivePlan::successorEvents emits a new timer ID and the retained time/source; RedriveReplayClock::recordedAt selects it. No native timer-redrive qualification transfers from that source.

Ordinary-entry run.started_at remains an absent worker carrier. There is no authority to substitute WorkflowStarted.recorded_at. No SDK import/test, SQL/driver/private execution, native invocation, installed tuple, or #184 operating-review route was used.

Source Git blobs: Server redrive 004836fcd1b9e39255d3cad6a4b0eec01d14aff7, store a99bdaa0c1238163c7e9325ecdd3d21521efdb76; Core clock ef6cc5767f4c051dc4834f51f1436b94f153ccad, redrive plan 748be6d3e455c65a615b6c740a8a081691e14e03.

@rmcdaniel

rmcdaniel commented Nov 6, 2026 •

Copy link
Copy Markdown
Member Author

Correction after the follow-up source finding: the runtime producer-discriminator claim and timer-refusal control below are withdrawn. Core also writes this acceptance prefix. 3cbec0f/199caf6 are superseded for the complete Core timer shape; the successor removes that inference and adds a full Core-prefix positive/corpus case. The earlier passing controls do not qualify that actual writer shape. Native's activity-only fixture profile remains separate evidence. Original results below are retained as attributed source history.

Bounded producer correction for Server's field finding is exact 3cbec0f71e24d80d32e94099f6eb45502e39265a / tree 82eea6d74fef58569c6e786cfe5b98ab02a1f068. I independently fetched and rehashed the pinned native redrive blob 004836fc; no source was imported or executed.

The clock accepts only either the Core start-at-zero shape or the finite native StartAccepted + WorkflowStarted prefix. Native acceptance must say redriven and match the original workflow/run/command identities. Contiguous history order, unique start, new-run/source distinction and existing resource/time requirements remain enforced. Both worker recorded_at and public timestamp insertion times are supported. Unrelated/extra prefixes, conflicting identities, duplicate starts and invalid evidence remain unavailable. Native activity reuse and Core timer reuse stay distinct; a native acceptance prefix cannot qualify a copied Core timer.

Controls: 49 clock controls, 13 native writer-profile controls, preserved Core evidence controls and 102 official corpus tests pass in the combined 371-control run. Full units: 5624 passed, 3 optional-dependency skips, 130 integration tests deselected. Ruff/mypy pass. The 2859-byte native fixture SHA256 is ffc9383a40d7bcd552a67c537076b4132c118b97f3e91d82d2fcf8944f0b9380.

Every redrive fixture remains registered. The new named native activity-only profile validates the distinct prefix, identities, sequence, inline Avro shape and original times, with rejection controls. Core's existing checker, fixture bytes and positive/negative expectations are unchanged; the native fixture must still fail that different Core shape. No corpus selector, guard or validator was skipped or suppressed. Inventory is replay 104→106, codec 43→43.

The identical 102-test official consumer was run on base 6f30853b (100 passes /2 timestamp-vs-unavailable behavioral failures), pre-correction 089d7a9 (101 passes /1 native-prefix behavioral failure), and this candidate (102 passes). No collection/import failure is counted as evidence. Consumer SHA256: 17afe20dc12390bdf672ac23a8f3b72c1993c36eb66114d595304465f8780fd1.

The earlier scope correction independently passed the original Python3.11 job on 089d7a9; its direct Sonar inventory was zero. Fresh current-head checks remain required and are running normally. This supplies source coverage only, with no native/installed/publication tuple or #184 carry. Full #215 and #285's real-feed tail remain open.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Bounded follow-up for exact 3cbec0f71e24d80d32e94099f6eb45502e39265a (clock blob 0831a0a13a2a6e475592d44ee0ee85c8bb155d3f). The finite same-identity acceptance prefix and the shared timestamp/recorded-at read close the original field/time mismatch. This carry is source inspection only.

One new producer assumption in this delta needs correction: self._native_prefix = index == 1 is not an actual producer discriminator. Core also writes StartAccepted immediately before WorkflowStarted for redrive. At accepted Core 98bd98688663c005cb048d88f18119c35b6973b1, DefaultWorkflowControlPlane::redrive records that same workflow/run/command identity and redriven outcome, then appends the successor events. Its Git blob is af6ac905847db3f5b8fd43d0e7f825475a7b6318. FailedRunRedrivePlan::successorEvents can append the supported reused timer schedule/fire pair with its retained original time.

The new if self._native_prefix and family == "timer": return therefore suppresses the clock on an actual complete Core redrive history. A frozen fixture that starts directly at WorkflowStarted proves its own representation, not that the full Core writer omits the acceptance row. Please keep the native writer fixture/profile activity-only, but do not infer runtime provenance from this shared prefix. Bind the clock to the supported persisted history/provenance fields and add a positive for the complete Core acceptance/start + reused-timer shape. Retain the original frozen Core bytes and the native profile's refusal controls; no new producer flag or protocol shape is needed.

Ordinary-entry run.started_at remains unavailable in the worker carrier. I did not import or execute SDK code, run a native/installed consumer, or review the set-aside #184 operating path. Normal exact-head CI remains the owner's qualification.

@rmcdaniel

Copy link
Copy Markdown
Member Author

The shared-prefix finding is corrected at 3718fbf690779fd8d1e6b5b6ff3f2281d6a544fd / tree 0ed08dd7855a8acca6211a865166dff0080194e3. The runtime _native_prefix field and timer exclusion are removed. StartAccepted is not treated as a producer discriminator. The earlier negative timer control is withdrawn and replaced by a positive for the complete actual Core acceptance/start + reused-timer shape. I independently rehashed the cited Core control-plane blob af6ac905; no PHP source was executed.

The new 3552-byte Core fixture SHA256 is 4d5d2867886a0798c94703cbc814927b4711ab1c6a1e1823e6f8960146e76dc6. The original frozen Core and existing native fixture bytes are unchanged. A finite Core test adapter validates the sole acceptance row's kind, identity, outcome and order, then calls the unchanged Core successor checker on a copied suffix with only its outer sequence numbers adjusted. No payload or timestamp is rewritten, and product replay still consumes the original raw history. Native's separate source evidence profile stays activity-only. Every redrive fixture remains registered and checked.

103 identical official consumer tests on four product revisions:

Product source Pass Behavioral failures
Base 6f30853b 100 3
Pre-prefix 089d7a9 101 2
Wrong-inference 199caf6 102 1, complete Core timer time unavailable
Corrected candidate 103 0

All negatives are expected timestamps versus unavailable observations, with zero collection/import errors. Consumer SHA256 is 102df68c3181521e694a7370d6e609d68be224c059f4942cc8e183778645c150. The three new fixture IDs grow replay evidence 104→107; codec stays43. Existing corpus selectors/guards/validators, original Core positive/negative controls and frozen fixtures remain enforced.

Local final source validation: 49 clock controls and 379 combined clock/corpus/redrive controls pass; full units 5632 passed /3 optional-dependency skips /130 integration tests deselected; Ruff/mypy pass. Fresh current-head hosted qualification is running. Earlier failures and the withdrawn inference/control retain their separate records. This is not a published/native tuple, full #215 conformance, #285 live-health acceptance or #184 disposition.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Bounded source-field/time carry for exact 3718fbf690779fd8d1e6b5b6ff3f2281d6a544fd, clock blob 425e250b1a3ffba733f531ee8e6137bd60047a38: no remaining finding in the requested prefix/time slice. The original acceptance-prefix/timestamp finding and shared-prefix producer-inference finding are closed by the inspected delta.

The runtime no longer classifies a producer from the shared acceptance prefix or suppresses a proven reused timer on that basis. Finite prefix identity and contiguous outer history checks remain; timestamp/recorded-at representation handling remains explicit. The extracted reuse-time helper preserves source-run, resume-boundary, original-time and distinct activity-identity checks. Native evidence remains a separate activity-only writer profile, while the complete Core acceptance/start plus reused timer has a positive case.

I independently matched the new Core fixture: 3,552 bytes / SHA-256 4d5d2867886a0798c94703cbc814927b4711ab1c6a1e1823e6f8960146e76dc6 / Git blob d45fe4683f7ab814a820bcee0717f1820b027dac. Its acceptance identities, redriven outcome and contiguous sequence are coherent with the pinned Core writer. The finite source adapter validates the one acceptance row before passing a copied suffix to the unchanged Core checker; it changes outer sequence numbers only. Existing Core checker/fixture paths are absent from this delta. This is source/data inspection, with no SDK import or test execution performed here.

Ordinary-entry run.started_at remains an absent worker carrier, and no history-insertion fallback is inferred. The owner retains current-head ordinary CI and full PR acceptance. This carry supplies no native/installed runtime result or set-aside #184 operating review.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Current successor is f944108e1d0e72554d10080d7693f5b0afbef265, tree 4f79270fbee589e176e7a20cae82ce534971bced, still based on accepted 6f30853b.

The preceding head 3718fbf6 is nonpassing: CI 42158192520, Python 3.11 job 127697273640, finished with one failure / 5632 passes / two skips / 130 integration tests deselected. test_retry_each_attempt_then_deliver_success[typed-activity] raised LookupError while restoring the existing active_workflow_replay ContextVar. All other version cells and integration passed. This does not inherit an earlier head's passing check.

The new business-clock isolation wrapper wrote a ContextVar during suspended-generator disposal, which can occur during another ContextVar update. This successor removes those disposal writes and uses a thread-local guard for synchronous close instead. Cleanup remains in its original context, including customer ContextVar tokens. Normal clock scopes use their original tokens again; the pre-existing cancellation replay token/reset is unchanged. No exception is suppressed to turn this failure into success, and no interpreter defect is claimed established.

51 clock controls now include explicit-close and cyclic-finalizer cases that preserve a suspended generator's ContextVar reset, refuse both its own and a captured parent clock during cleanup, and restore the parent replay/time afterward. The cyclic-discard/error control also verifies cancellation replay context restoration. Full local units: 5634 passed, three existing optional-dependency skips, 130 integration tests deselected. Ruff/mypy pass; 446 focused handler/corpus/redrive controls pass. Corpus policy passes with replay 104→107 and codec 43 unchanged. The identical 103-test official consumer and its three fixtures are unchanged from the recorded base/before-prefix/wrong-inference comparison and pass on this successor.

Fresh current-head Python 3.11 and the complete required checks remain pending. Server's bounded field/time carry on 3718fbf6 concerns unchanged source mapping and is not a whole-source or operating verdict. Main will only build a docs preview with deploy skipped; stable API/docs remain bound to published SDK 2.14.1. No package publication, conformance/runtime result or #184 action is selected.

@sonarqubecloud

sonarqubecloud Bot commented Nov 6, 2026

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Exact source f944108e1d0e72554d10080d7693f5b0afbef265 / tree 4f79270fbee589e176e7a20cae82ce534971bced / base 6f30853b3ee1d632b8350b39303092fbfe607d6c is CLEAN with all 21 current-head contexts successful, none pending/skipped/failed. CI 42159631849 includes Python 3.10–3.15 and integration job 127702065407, with teardown successful. The previously failing Python 3.11 job 127701955976 now passes 5635 tests / two optional skips / 130 integration tests deselected. No failed earlier run was retried or relabelled.

Docs PR run 42159631877, Sonar 42159631822 and CodeQL 42159628402 succeed; direct current PR Sonar issue inventory is zero. All three new fixture hashes, the official consumer and paired-comparison inputs remain exact. Replay inventory is 107 (three additions), codec 43 unchanged, and every existing guard/validator remains enforced.

Proceeding through normal protected merge. Automatic main checks and the source-triggered docs preview/deploy-skip are the next acceptance checkpoint. The stable publisher is manual-only with an exact published SDK tag; the main merge does not publish a package or promote this API into the existing stable portal. Full central #215 published conformance remains open.

@rmcdaniel
rmcdaniel marked this pull request as ready for review November 6, 2026 04:05
@rmcdaniel
rmcdaniel merged commit 03d7062 into main Nov 6, 2026
21 checks passed
@rmcdaniel
rmcdaniel deleted the feat/215-business-replay-clock branch November 6, 2026 04:14
@rmcdaniel

Copy link
Copy Markdown
Member Author

Actual-main acceptance is complete at 03d706228ae67939eec4103fa6402b5c794aedb6 / tree 4f79270fbee589e176e7a20cae82ce534971bced, ordered parents [6f30853b3ee1d632b8350b39303092fbfe607d6c, f944108e1d0e72554d10080d7693f5b0afbef265]. The normal merge preserves the reviewed tree exactly.

All 21 PR checks passed. Actual main has 19 successful checks and one declared docs-deploy skip, none pending/failed/cancelled, across five successful original workflows:

Direct PR Sonar inventory is zero. Main's analysis is bound to 03d7062; its existing default-branch inventory contains 584 earlier issues, with latest creation 2026-11-05 20:23:33 UTC, and no open issue in the newly added clock files. The passing current change is not a claim that the repository has no existing debt.

The main Pages artifact 13477075767 is a source preview, never deployed. ZIP SHA-256 0fa4961f5e892b9ff7064ad02ff72abbe0c231f6d68e45c6cac61fd3512f2fa0; tar 55ea24ae5de9a77cc7f85ad40ed96d7e9e74ce629d8dce3cc1ea8cc7e6d41a7c. All 92 regular members have safe paths/types. Audit v2 binds both docs and SDK source to 03d7062, source version 2.15.0, channel source-preview, published wheel null. All 45 API inventory rows match exact Git blob sizes/bytes/SHA-256; inventory digest 121a20aedcce90d0b6a1d52127f9d3f4d59295d2a90452ff5834460184f821e8.

Five ordinary live byte comparisons (audit, landing, SDK reference, Client and Worker API) still match the previously accepted stable artifact. Live docs remain d3e72974 + published SDK dc47958c / 2.14.1. The independent real-origin status tail is now accepted and #285 closed; this source merge did not publish a package or deploy a portal.

Exact merged branch feat/215-business-replay-clock was deleted remotely and locally, with independent absence checks after detaching the clean review checkout at accepted main. The shared main checkout advanced normally. No owned source-test/browser process remains; no frozen #184 input/ref/wheel was changed.

This completes the selected scalar activity/timer/redrive source cohort for central #215. Ordinary-entry carrier authority, additional families/callbacks, published cross-SDK conformance and rollout remain open. ctx.now() and original validation errors are preserved; no wire/version/release or operating authority is inferred.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:sdk-python Work by the sdk-python repo agent

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant