Repository navigation
fix(durable): don't journal a step whose context ended partway - #234
Merged
Merged
Conversation
4 of 5 tasks
reka18
force-pushed
the
fix/durable-canceled-service
branch
from
November 5, 2026 16:38
15ab425 to
41bfbb4
Compare
reka18
force-pushed
the
fix/durable-canceled-service
branch
from
November 5, 2026 16:40
41bfbb4 to
f87bc98
Compare
reka18
force-pushed
the
fix/durable-canceled-service
branch
from
November 5, 2026 16:53
f87bc98 to
8c4ce15
Compare
reka18
force-pushed
the
fix/durable-canceled-service
branch
from
November 5, 2026 17:09
8c4ce15 to
83c5655
Compare
reka18
force-pushed
the
fix/durable-canceled-service
branch
from
November 5, 2026 17:14
83c5655 to
1023215
Compare
reka18
force-pushed
the
fix/durable-canceled-service
branch
from
November 5, 2026 17:22
1023215 to
1890a2c
Compare
reka18
force-pushed
the
fix/durable-canceled-service
branch
from
November 5, 2026 17:33
1890a2c to
2e345fa
Compare
reka18
force-pushed
the
fix/durable-canceled-service
branch
from
November 5, 2026 17:42
2e345fa to
96cceff
Compare
reka18
force-pushed
the
fix/durable-canceled-service
branch
from
November 5, 2026 18:04
96cceff to
0d6f9ce
Compare
reka18
force-pushed
the
fix/durable-canceled-service
branch
from
November 5, 2026 18:20
0d6f9ce to
f6c246d
Compare
…s only on a refusal A respawned actor's start stays quiet, but the parent Fires its failure or settle drives (onError, onDone) and the effects they absorb now run as under Absorb. Before, an actor such a Fire spawned couldn't reach the parent, and its messages were dropped without being journaled. Start fails only when a context's ending refused a held actor Fire. A context that ends with no Fire refused no longer fails a consistent Start, which a retry couldn't recover from (ErrInstanceExists). Also: Recover absorbs the running and retried spawns in one call each, the service and actor start filters share one helper, and heldFires is inlined. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…the system AbsorbRespawn kept a respawned actor's messages from the parent, but let through the onDone or onError of a nested actor its start spawned. The journal holds that Fire too, so Recover applied it twice. Now only the actors AbsorbRespawn names can drive a new parent Fire: a spawn that fails now, or one that settles while the others start. While such a Fire and its effects run, the parent hears them. The quiet mark is a field on the ActorSystem, not a context value, so a dispatch to the parent no longer walks the context chain, and the mark can't leak into a context a behavior captures. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
A parent Fire an actor drove (its onDone, onError or a message) reached only the actor system, so the state it entered had no service in flight and no timer armed, live or after Recover; a checkpoint then recorded the service as settled. Such a Fire now arms them, live, at Start and Recover, and on replay, as a host Fire does. Fixing it exposed clock-read gaps, now closed: DeliverToActor and RunService journal their step's clock reads; a replayed service settle re-arms its timers; Recover keeps its spawns' reads and re-arms a checkpoint's timers before it spawns; Tick drops the timers it fired at its own read. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
During AbsorbRespawn, a nested actor whose spawn failed, or that crashed, with no onError escalated again on every Recover, though the live run had already reported it. Escalation now follows the quiet rule: a failure the journal's run already saw stays quiet, and a respawned actor's own new failure still escalates. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
… old journals as before - A checkpoint records the services that settled, not every declared service that isn't pending. A service a timer's state invoked never started, and Recover used to skip it for good. Re-entering or leaving a state forgets its settle. - Tick drops each timer the scheduler fired, by id, live and on replay. A timer that came due between a tick's clock reads, or a chained zero-delay timer, no longer stays in the table to fire again after Recover. - A journal written before actor-driven Fires armed timers and services replays without arming them, so its clock reads line up as they did. - An actor-driven Fire's arming gets the step's context; actor sets are computed each step; Recover builds one replay mirror. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…urnal format Recover re-armed a timer the checkpoint recorded even when a later step had canceled it, since the cancel replays before the re-arm. The replay mirror now notes the timers the tail cancels, and the re-arm skips them. Each Record now carries Format, and replay arms an actor-driven Fire's timers and services from it, instead of inferring the format from whether UnstartedActors is nil. The CHANGELOG tells Store authors which fields to keep. Also narrow the CHANGELOG's context-error sentence to Fire callbacks, correct the DeliverToActor godoc for a delivery that drove no parent Fire, and drop dead code: replayed.tick, the unused receivers on stepCanceled and consumedCanceled, and actorEntry's parameters that repeat the ParentFire. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…y a refused onError Recover restored a checkpoint's pending timers only after the tail walk, so a tail Tick that fired one live fired nothing on replay, and the timer was re-armed and fired again. Recover now arms them into the replay scheduler at their persisted deadlines before the walk, through state's new Scheduler.ArmAt, which takes no clock reading. Tail cancels and Ticks then act on them as they did live, which replaces the canceled-timer filter. A spawn whose onError a guard or middleware refused counted as unstarted, so every Recover ran its behavior again. It now counts as handled. A spawn with no onError to route to, or whose onError a context's ending refused, is still retried. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
… checkpoint timers to #236 An invoked actor with no onError fires the zero event at the parent when it can't start, so durable could tell "none declared" from "a declared onError was refused" only by matching the refusal's text. That retried a declared onError the parent had no transition for, and a denial worded the same way. state now reports it: SpawnActor.NoOnError and ParentFire.NoOnError, additive. The builder records the declaration, so an onError that is the zero event, such as an iota enum's first constant, still counts as declared. The zero event still fires, as v1 documents. Recover retries a failed spawn only when it declared no onError or a context ending refused its onError. Restoring a checkpoint's timers before the replay walk is #236's, so this takes that change and its tests back out, along with state's Scheduler.ArmAt, and Recover re-arms them after the walk as before. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…in review questions with tests A spawn that failed with no onError declared counted as uncaught even when the parent handled the zero event it routes, so Recover retried it, running the behavior again. An onError Fire that applied now always counts as caught; NoOnError only decides what a refused zero event means. Tests that settle questions from the review: - a re-entry spawn retried at Recover doesn't repeat the first start's message; - a lifted Fire that re-enters a state during AbsorbRespawn starts its actor once; - a timer leaving a service state, and a timer entering an actor state, leave the drivers as live does. Skipped: Tick hands a timer transition's effects only to the scheduler, which #236 fixes. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…eduler armed it at Tick recorded a chained timer's deadline from the tick's last clock read, later than the absorb read the scheduler armed it at, so a checkpoint persisted a later deadline and the timer fired late after Recover. Scheduler.Tick reads the clock once to find each due timer and once to absorb its effects; Tick and its replay now take the i-th fire's absorb read. A state test pins that order of reads, so a change to it fails there rather than drifting deadlines silently. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…nError An InvokeActor actor with no WithInvokeOnError fired the zero event at the parent when it couldn't start or failed after running, and a wildcard, a forbid or a transition on that event could take it as a declared onError. Durable then counted the spawn as caught and never retried it. The actor system now reads SpawnActor.NoOnError and escalates, as it does for a Spawn with no onError, firing nothing; durable retries such a spawn at Recover. ParentFire.NoOnError, which no Fire could set any more, is removed. Tick and its replay now fail when a fired timer's arming read is missing, instead of falling back to another read, so a change to the scheduler's order of reads surfaces. Two comments describing the old deadline rule are corrected. A parent Fire an actor drove doesn't have its domain effects dispatched; its test is skipped for #235. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
Working the deadline out from the order of the Tick's clock reads breaks when a fired transition reads the instance's clock itself: the extra read moves the index, and the recorded deadline silently differs from the scheduler's. state's Scheduler now reports a pending timer's deadline (Scheduler.Due, additive), and Tick and its replay take each chained timer's deadline from it. The read-order helpers go. Tests: Scheduler.Due; a chained deadline when the fired transition reads the clock; an uncomparable event in a JSON-loaded onError doesn't panic declaresOnError. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…fail on a lost Format - state: AbsorbRespawn's quiet rides the call chain that spawns each actor, rather than one flag for the whole system, so a delivery another goroutine makes while it runs is heard. It ends when AbsorbRespawn returns. - state: an Invocation writes the onError key only when one was declared and reads a present key as declared, so a zero-valued onError survives the JSON IR. IR written before this always has the key and keeps routing as it did. - durable: the Start baseline and checkpoints record the journal format, and replay fails on a Record whose format is lower, instead of silently replaying a Store that lost Record.Format as an older journal. - durable: the actor sets are reused while the configuration, the running actors and the step's spawns are unchanged. - durable: actor entries are decoded once on replay; actorBuf, the dropFiredTimers wrapper, the replayed struct and a redundant nil check are gone. Tests confirm each: a delivery during AbsorbRespawn is heard; a zero-valued onError survives JSON; Recover fails on a Store that drops Format; a step that touches no actor allocates about as much with actors as without. A host read of the instance clock mid-step is journaled but can't be replayed; its test is skipped pending an issue, as it isn't this PR's. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…over from an older checkpoint An older version started only the active leaves' services and actors at Start and Recover, so its checkpoints record no compound state's as settled or ended. Now that StartEffects includes a compound state's, Recover from such a checkpoint put a service that had settled into a child back in flight, and spawned an actor that had finished. For a checkpoint without a journal format, Recover counts a compound state's services as settled and its actors as ended, as that version left them, including on a later Recover after this version has recorded steps on top. The skipped host-clock test now points at #247. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…e its spawns ran - durable: DeliverToActor records nothing and fails the Handle once its context ended by the end of the delivery, even when it drove no parent Fire. The ending may have made the kernel refuse the actor's own Fire, losing the event, while the call reported success. - durable: Recover no longer fails on a context that ends once it has spawned its actors with no Fire refused, as Start doesn't; their behaviors already ran, and a retry would run them again. - state: ResumeEffects covers the active leaves alone again, services and actors as well as timers. After a Restore, a compound state's service or actor may already have settled into a child, and starting it again ran it twice. - state: an Invocation's JSON is parsed once on load; AbsorbRespawn's doc no longer warns against concurrent deliveries, which the scoped quiet hears. Tests: a delivery refused inside the actor; Recover's context ending during a respawn; ResumeEffects after a compound service settled; every Invocation field kept when it declares no onError (by reflection, so a later field is covered). Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…n undeclared invoke - durable: Recover checks its context once more after the replay walk, before any actor spawns. b97e07d dropped the only check after the last Record, so a context that ended while the last Tick or settle replayed could return a Handle short of the journal. - state: an invoked actor with no onError emits its SpawnActor with a nil OnError, as a Spawn with none does, so a host that reads the effect sees none declared; the actor system already escalates on a nil OnError. SpawnActor.NoOnError, which only stood in for that, is removed. - durable: DeliverToActor, Start and Recover restore the actor collector's step context in a defer, so a panic out of host code, such as a parent middleware's, doesn't leave it driving under the dead step's context. Start and Recover build StartEffects once. - state: the ResumeEffects and StartEffects docs say which to call after a Restore. Tests: the context ending during the last replayed Record; an undeclared invoke's nil OnError; a middleware panic during a delivery. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…tled Pins that an actor restored from a snapshot resumes without spawning again a nested actor its earlier start spawned, once that nested actor settled and moved it on, for a nested actor declared on the actor's leaf state and on a compound state. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
This was referenced Nov 16, 2026
Open
…d onErrors as issues Reverts three changes to state's v1 behavior, so this PR adds to state without changing what it does: - StartEffects covers the active leaves again, as v1 does. A compound state's services and actors don't start at Start or Recover; tracked in #249. Recover's handling of an older checkpoint's compound invokes, which only that widening needed, goes too. - An InvokeActor with no onError fires the zero event again, as v1 documents, rather than escalating. durable tells "none declared" from "refused" by the kernel's refusal reason, a stopgap with known gaps; tracked in #250. - An Invocation's JSON writes onError as before; a declared zero-valued onError is lost through the IR, tracked in #251. The tests for what those changes fixed stay in the PR, skipped with a pointer to the issue. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
reka18
commented
Nov 16, 2026
reka18
left a comment
Contributor
Author
There was a problem hiding this comment.
Pointers to the issues this PR leaves open, at the code they touch.
- InvalidTransitionError.Is matches ErrNoTransition only for the kernel's own refusal - durable drops its copy of the reason-string check - a host refusal worded like the kernel's now counts as handled; unskip its test
- compare []S with slices.Equal instead of fmt.Sprint per step - idle step with three actors: 35 to 32 allocs, 4088 to 4032 B/op - add BenchmarkHandle_Fire_ActorIdleStep
…tarted set - Mark Recover's quiet respawns so a failed one stays in neither actor set - Next Recover respawns it quietly, so its start message applies once - Cover zero-event, no-onError and context-ended respawn failures
Collaborator
|
Thanks for this one, it's a careful piece of work. I ran a few review passes and pushed three small commits on top. Happy to revert any of them if they don't fit your plan. Pushed
Looked at, no change
Checks
|
joshua-temple
approved these changes
Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this change does
A step whose context ends partway no longer journals what the kernel may have refused, and recovery now replays actors, services and timers the way the live run ran them.
One rule for an ended context.
Tick,RunService,DeliverToActorandFirerecord nothing and fail the Handle withErrHandleFailedonce the step's context ended by the end of the step, or once a Fire the step drove was refused with a context's error. The second case catches a context middleware derived from the caller's.Fire,StartandRecovercheck only for the refusal, throughrefusedByContext, since a context that ends after their Fire applied cost them nothing.Tick,RunServiceandDeliverToActorcheck the context too, throughstepCanceled;Tickonly when it fired a timer.Tick: records nothing once its context ended, even for timers that applied.Recoverre-arms the timers, and they fire again.RunService: records nothing once its context ended during the service or while the settle's effects were absorbed. AfterRecoverthe service is in flight again and runs a second time.DeliverToActor: records nothing once its context ended by the end of the delivery, since the ending may have made the kernel refuse the actor's own Fire and lose the event. AnInvokeActoractor restarts from its initial state afterRecover.Fire: records nothing when an actor its state spawned failed to start and a context's ending refused that spawn's onError. The event applied, so fire it again on the recovered Handle.StartandRecover: fail when a context's ending refused a parent Fire one of their actors drove, such as a failed spawn's onError. A context that ends with no Fire refused doesn't failStart, orRecoveronce it has spawned its actors, since their behaviors already ran. The baseline is recorded, soRecoverthe instance.Recoverfail.Actors replay as they ran.
Recoverretries a spawn that failed to start when it declared no onError, or a context ending refused its onError. One whose declared onError applied, or was refused for any other reason, such as a guard returning false, is done: the journal holds the refusal, and a retry would run the behavior again. An invoked actor with no onError fires the zero event at the parent, as v1 does, sodurablereads a "no transition" refusal as none declared. That's a stopgap with known gaps, tracked in state: an InvokeActor with no onError fires the zero event at the parent instead of escalating #250.Record.EndedActors, and those that never started inRecord.UnstartedActors; checkpoints keep both.Recoverspawns only the declared actors that aren't ended.Start's andRecover's spawns, they're journaled with the step after. Replay re-fires each where it ran.Recoverspawns the actors that were running withstate's newActorSystem.AbsorbRespawn, before the actors whose spawn it retries. Nothing their start drives reaches the parent again: not their messages, not a nested actor's onDone or onError, and not an escalation the live run already reported. A respawned actor that fails or settles now, which didn't happen live, still reaches the parent.Services and timers follow actors.
StartandRecover, and on replay, as a hostFiredoes.Recoverdoesn't start them again. Re-entering or leaving the state forgets the settle.Tickdrops each timer it fired, by id, live and on replay, so a fired timer can't stay in a checkpoint and fire again afterRecover.DeliverToActorandRunServicejournal their step's clock reads, so a checkpoint on that step can't leave a read for the next step to replay.Tickand its replay take it fromstate's newScheduler.Due, so a later checkpoint persists the deadline the scheduler armed, whatever clock reads the tick's transitions took.Also:
RecoverandStateAtsharereplayRecord. Both fail on a context that ends during the walk, checked before each replayed Record.ci.yml, the requiredgateincluded, skips on a draft. The cost is stated inci.yml's header: a skipped required check counts as passing, so a PR merged in the seconds between marking it ready and its new run starting would merge without CI. That window is accepted to keep drafts free of runs.stateadditions (additive, v1.1.0):ActorSystem.WithParentFireObserverreports each parent Fire the system drives as aParentFire: its result, the actor, whether it settled that actor with its output or error, and whether it routed a failed spawn's onError.ActorSystem.AbsorbRespawnspawns actors that were running without letting their start reach the parent again. Its quiet rides the call chain that spawns each actor, so a delivery another goroutine makes meanwhile is heard.Scheduler.Duereports a pending timer's deadline.Journal compatibility. Each Record, the Start baseline and each checkpoint now carry
Format, 1 from this version. Replay arms an actor-driven Fire's timers and services only for a Record of format 1 or later, and fails on a Record whose format is lower than its checkpoint's, which means the Store lost the field. A journal this version writes isn't for an olderdurable, so don't downgrade a Store it has written to.A journal an older version wrote replays as it did for its actor-driven Fires, which arm nothing. A few replay changes apply to it with no version check, each closer to what the older live run did: a replayed settle arms its state's timers, a replayed
Tickdrops each timer it fired, and settles count toward the next checkpoint's settled services.For Store authors: a Store must keep
Record.Format,EndedActorsandUnstartedActors, and keep an empty actor set distinct from nil. MemStore and FileStore do. A Store that dropsFormatnow failsRecoverinstead of recovering wrong.Why
The kernel refused a step live, but the result was journaled, so recovery committed something the live instance never applied. For an actor, it left a journal
Recovercouldn't replay. The seams guarded this inconsistently, and none saw a refusal from a derived context. Fixing that exposed how recovery treated actors: it spawned actors the live run had ended, lost or doubled the parent Fires their start drove, and gave outputs to the wrong transition. Journaling those Fires in turn exposed that a state an actor entered never armed its timers or services.Relates to: #214, #243, #245
Left for a follow-up:
Recoverrestores a checkpoint's timers after the replay walk, so a tailTickmisses them, and a tail step that cancels one doesn't keep it canceled. fix(durable): keep timers across recovery and skip idle ticks #236 restores them before the walk.Tickhands a timer transition's effects only to the scheduler, so a timer that leaves a state doesn't stop its services, and one that enters a state doesn't spawn its actors. fix(durable): keep timers across recovery and skip idle ticks #236 takes this on; the tests are here, skipped.Instance().Clock()during a step has that read journaled, butRecovercan't make it again, so later replayed reads shift. Tracked in durable: a host read of the instance clock during a step is journaled but can't be replayed #247; the test is here, skipped.StartandRecoverstart only the active leaves' services and actors, not a compound state's. Tracked in state: StartEffects leaves out a compound state's services and actors, so Start and Recover don't start them #249; the tests are here, skipped.InvokeActorwith no onError fires the zero event rather than escalating, and the JSON IR can't tell a declared zero-valued onError from none. Tracked in state: an InvokeActor with no onError fires the zero event at the parent instead of escalating #250 and state: the JSON IR can't tell a declared zero-valued onError from none #251; the tests are here, skipped.Recoverrespawns running actors in configuration order, not the order they spawned live. A respawned actor can then receive a message it missed live, and if that settles it, its onDone or onError reaches the parent. Tracked in durable: Recover respawns running actors in configuration order, not live spawn order #246.Evidence
On
GOTOOLCHAIN=go1.27.4, at the PR head:go test -race -count=1 ./...in every workspace modulego vetondurableandstategolangci-lintondurableandstateNew tests cover:
Startwith and without a refusal;Recoverand checkpoints;Fire,RunService,StartandRecover, each applied once;Recover;Tickfired between its clock reads, and a chained zero-delay timer;Format, replayed without arming, and a Store that dropsFormatfailingRecover;ResumeEffectsafter a Restore not restarting a compound state's settled service;Recoverfailing when its context ends during the last replayed Record, and returning its Handle when it ends once its actors are spawned;AbsorbRespawnhearing another actor's delivery, and a lifted Fire that starts a re-entered actor once;Recover;Each fix fails its test with the fix removed.
Checklist
git commit -s) per the DCOtype: subject)mage checkpasses locally (lint + race tests + govulncheck): not run as a whole; the parts above were run individually