Repository navigation
Conversation
4 of 5 tasks
reka18
force-pushed
the
fix/durable-effect-dispatch
branch
from
November 5, 2026 16:38
ebb5863 to
ef9c0a8
Compare
reka18
force-pushed
the
fix/durable-effect-dispatch
branch
from
November 5, 2026 16:40
ef9c0a8 to
d46adf2
Compare
reka18
force-pushed
the
fix/durable-effect-dispatch
branch
from
November 5, 2026 16:53
d46adf2 to
cc4d8cc
Compare
reka18
force-pushed
the
fix/durable-effect-dispatch
branch
from
November 5, 2026 17:09
cc4d8cc to
d657891
Compare
reka18
force-pushed
the
fix/durable-effect-dispatch
branch
from
November 5, 2026 17:14
d657891 to
3e41947
Compare
reka18
force-pushed
the
fix/durable-effect-dispatch
branch
from
November 5, 2026 17:22
3e41947 to
3edd2ee
Compare
reka18
force-pushed
the
fix/durable-effect-dispatch
branch
from
November 5, 2026 17:33
3edd2ee to
aae543d
Compare
reka18
force-pushed
the
fix/durable-effect-dispatch
branch
from
November 5, 2026 17:42
aae543d to
50bbc98
Compare
reka18
force-pushed
the
fix/durable-effect-dispatch
branch
from
November 5, 2026 18:04
50bbc98 to
a14c4ed
Compare
reka18
force-pushed
the
fix/durable-effect-dispatch
branch
from
November 5, 2026 18:20
a14c4ed to
90bcbfb
Compare
Contributor
Author
|
Review results. One reviewer per dimension, each reading the others' findings. All seven dimensions have zero blockers on the latest round. Fixed along the way: For a maintainer: questions 3 and 4 on #214 still need a ruling. The description says where this PR departs from the proposed answers. |
This was referenced Nov 9, 2026
Only Fire dispatched domain effects. Tick and RunService now record a step through the same helper as Fire, then dispatch its effects as Fire does, after the append and before any checkpoint. A failed dispatch fails the Handle before its checkpoint, so a later checkpoint can't compact the effect away, and Recover retries it. Start applies the initial entry's effects before the baseline, so a handler failure leaves no instance. EffectInstance gives a handler its instance id, and state.ServiceRunner gains WithResultTransform so a service's result reaches the machine in the form a replay supplies. Effects from a parent transition an actor drives are still not dispatched. Relates to #214. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…s a replay can read Review follow-ups for the effect dispatch change, each pinned by a test. - One helper, applyEffect, calls the effect handler for Start, a step and Recover, and turns a handler panic into ErrEffectDispatch. Start and Recover used to let the panic out, for a Fire's effects as well as a tick's or a service settle's. - A service result whose JSON form won't decode, such as a number beyond float64, now fails RunService with ErrHandleFailed and records nothing, as a result that won't encode does. It used to be journaled, and every Recover then failed to decode it. - A service error whose Error method panics settles with "service error panicked: ...", its invalid UTF-8 replaced, so live and replay see the same message. encodeEventData and replayableOutcome share normalizedErrorMessage, which drops replayableOutcome's dead fallbacks. - Recover re-dispatches a step's effects only when its Record names them, so it doesn't send effects from an older version's Tick or service steps, which never dispatched them. - state: a WithResultTransform that returns no error for an unbound service no longer turns it into a success; the service still settles through onError. - Tests that Recover re-dispatches a tick's, a multi-timer tick's and a service settle's unmarked effects once, under the ids the live run stamped. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…in Start's at-least-once effects A service result now decodes through the Runner's EventDataDecoder for the invocation's onDone event, live and on replay, as a WithEventData payload does. A decoder for that event hands Assign its type; without one the result arrives in JSON's generic form. A service error stays message-only. - state: ServiceRunner.WithResultTransform's function takes the event that will carry the outcome, func(event E, result any, err error) (any, error), and runs on every settle: Tick, SettleDone and SettleError. It runs before any event fires. An UnboundRefError still settles through onError when the transform returns no error. - A decoder that fails on a result, or returns a type other than the service returned, fails RunService with ErrHandleFailed wrapping ErrEventDataEncoding before any onDone or onError Assign runs, and records nothing. Recover and StateAt fail with ErrEventDataEncoding on a recorded result their decoder can't decode, also running no Assign. This also covers a result whose JSON form won't decode, such as a number beyond float64. - The docs say to register the decoder before results are recorded: replay decodes the journal with the recovering Runner's decoder. - Tests pin the decoding scenarios, that a decode failure runs no Assign, and the documented hazard of a decoder added after recording. - Tests pin Start's initial effects as at-least-once: applied before the baseline write and never marked, so a handler failure, a failed baseline write or overlapping Starts re-apply them, and Recover never applies them. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…icking ServiceRunner.WithResultTransform's function now returns an explicit abort: func(event E, result any, err error) (out any, outErr error, abort error). - A non-nil abort stops the settle: no event fires, so no Assign, middleware, inspector or history entry. SettleDone and SettleError return false with the abort in FireResult.Err; Tick returns that FireResult as its one element. LastResult and LastError keep the previous settle. - The transform runs before the service is dropped, so an aborted settle, or a transform that panics, leaves the service in flight. A panic in the transform goes on out of the settle, as any host panic does. - durable's transform returns an abort for a result that won't decode for its onDone event, in place of the resultDecodeFailure panic. RunService and replay check the returned abort; tickService, settleRecorded and their recover() are gone. Live, the step still fails the Handle and records nothing, and Recover has the service in flight again; replay still fails with ErrEventDataEncoding. - TestWithResultTransform_AbortFiresNoEvent replaces the panic test, covering Tick, SettleDone and SettleError. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
… results Each finding gets a test that fails if it's real, committed whether it proved or disproved the finding. - state: a settle can't settle a later activation of its service. Each activation carries a generation; a SettleDone or SettleError whose transform ran, or a Tick whose ServiceFn ran, while the service was stopped and armed again under the same invoke id now returns false and leaves the new activation in flight. - durable: a DispatchStore whose Dispatched or MarkDispatched panics fails Recover and the stateless Runner.Fire with ErrEffectDispatch, as a step already did. The guard moves into the shared dispatch path, which applyStartEffects now shares its handler and DispatchStore checks with. - durable: a decoder may return json.Number, JSON's generic form under UseNumber, for any service result. The service and Fire paths share one type check; a Fire payload stays strict. - durable: DeliverToActor records through recordStep and finishStep; persistStep is gone. - Pins: an undecodable service result reruns the ServiceFn on every Recover; an unmarshalable KindedEffect fails Tick and RunService with no handler wired; Recover loads the dispatched set once per effect Record, as main did for Fire Records; the journal holds the decoded result's encoding, which differs from the first encoding for a generic struct or an int64 above 2^53. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…ct that won't marshal by id and kind - durable: a service result the decoder for its onDone event refuses, or decodes as another type than the service returned, is parked. RunService returns ErrEventDataEncoding, not ErrHandleFailed, before any Assign runs, and records the result with its Go type in a Record that settles nothing. The Handle stays healthy and the service stays in flight, but its ServiceFn never runs again: replay parks the result again, and a RunService once the decoder decodes it settles the parked result, held to the type the service returned. The park step offers its checkpoint as any step does, and the checkpoint envelope carries parked results in a new parkedServices field, so compaction can't lose one. Leaving the state that declares the service discards the parked result; entering it again runs the ServiceFn for the new activation. StateAt replays a parked Record as a step that changed nothing. state's ServiceRunner is unchanged: a parked service is in flight there. - durable: a KindedEffect whose payload won't marshal is recorded by id and kind with no payload, as an effect that isn't a KindedEffect is, with the marshal error's message in Meta under crucible.marshalError. No step fails on it, Fire included, and it dispatches as any effect does. - The pins for both now assert the new behavior: the ServiceFn runs once, and every kind of step records the effect. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
… format Each finding gets a test that fails if it's real, committed whether it proved or disproved the finding. - durable: a ctx that ended during the ServiceFn parks nothing. A result the decoder refuses fails the Handle and records nothing, as a result that decodes does, and the service runs again after Recover. - durable: the journal records the JSON the live settle decoded, not the decoded value encoded again, so replay decodes the same bytes even for a decoder type whose MarshalJSON and UnmarshalJSON aren't inverses, or whose decoded value won't encode again. The second marshal is gone. - durable: journalFormat is 2. A service result in a Record of format 0 or 1, which an older version wrote when live saw the ServiceFn's own value, replays in JSON's generic form through no decoder, as that version replayed it, so a decoder added for its onDone event can't fail Recover or StateAt on it. - durable: a parked result records its type with its import path, so a decoder for another package's type of the same short name can't settle it. - durable: a result with no JSON form aborts the settle, so no Assign runs, and fails the Handle as before. - durable: a KindedEffect whose MarshalJSON panics is recorded by id and kind with the panic's message under crucible.marshalError, rather than panicking out of a step the kernel applied. A Kind that panics still fails the Handle; a test pins it. - durable: replay decodes each service payload once, and typeName runs only for a park. parkedServices dedupes with a sorted set, as endedServices does. - docs: the Handle that parked a result never sees a fixed decoder; every doc site now says to Recover with the fixed decoder, then call RunService. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
Recover, the stateless Runner.Fire, StateAt and Steps now refuse a journal whose checkpoint or any Record records a journal format above this version's, before they restore or replay anything. The check covers a checkpoint with an empty tail, so each later bump of journalFormat protects a rollback. Formats 0, 1 and 2 still load. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…ixed versions and stale settles Each finding has a test that fails if it is real, kept whether it proved or disproved it, and each fix was checked by reverting it. - Recover sends what replaying a step produces by the Record's format, not its Effects envelopes: a Fire's always, a tick's or a service settle's from format 2. A Store that drops Record.Effects no longer loses an effect a crash left unsent; an older version's tick and service effects still aren't sent. - Recover loads the dispatched set once, not once per Record with effects: 1000 tick Records go from about 22 ms to 2 ms on MemStore, 25 ms to 4 ms on FileStore. - A DispatchStore error in Dispatched or MarkDispatched wraps ErrEffectDispatch, as a panic there already did; the ErrEffectDispatch doc now covers the dispatch tracking. - The Record that settles a parked result records its type, and replay holds the decoder to it, as the settle did. - A parked result's type name keeps the import path inside struct literal, func, chan and interface types, with struct tags and unexported fields' packages. - A Record of a lower journal format after a newer checkpoint is reported as an older version appending to a newer journal; a Record with no format still blames the Store. The CHANGELOG says not to run mixed versions on one instance. - A settle that finds its activation stopped or armed again records nothing. The step lock keeps durable's own steps from causing it; an internal test drives the ServiceRunner directly. - The JSON a settle decoded is cleared once the step, or Recover's replay, has used it. - Settled services and parked results are updated together through one helper on the Handle and one on the replay mirror. - Pins: a moved or renamed type leaves a parked result parked under a typed decoder, and JSON's generic form still settles it; a ServiceFn can't change its own activation through the Handle. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
This was referenced Nov 17, 2026
Open
…'s type, reread the dispatched set before unmarked effects Each finding has a test that fails if it is real, kept whether it proved or disproved it, and each fix was checked by reverting it. - A Record of a lower journal format than any before it, the checkpoint's or an earlier Record's, is refused as an older version appending to a newer journal, not only one below the checkpoint's. Recover and StateAt check every Record's format before they replay any, so a refused journal sends no effect. - Every successful service settle records the Go type the service returned, and replay holds the decoder to it, as for a settled parked result. JSON's generic form and nil are always accepted, and a Record of format 0 or 1 still replays with no check. Changing the type a decoder returns for an event with recorded results now fails Recover unless it returns the generic form; the RunService doc, package doc and CHANGELOG say so. - Recover reads the dispatched set again before sending a step's effects that its copy doesn't show marked, so a mark a stale owner makes during a handoff is seen, as the per-step load did. A step whose effects are all marked costs no read: 1000 landed tick effects still take one read, and about 2 ms on MemStore. - The tick-tail benchmark also measures a tail with no effect landed, and its run line names it correctly. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…n effect handler on a Store without dispatch tracking - state: ServiceRunner.Tick returns an empty slice when the activation its ServiceFn ran for was stopped, or stopped and armed again, before the settle, as for an id not in flight. An abort still returns its one-element slice with the abort in Err. A non-empty slice now always means the settle fired or was aborted, whatever FireResult the Fire returned. - durable: serviceStep reads a stale settle from Tick's empty result, and settledNothing, which guessed it from a zero FireResult, is gone. A middleware that returns a zero FireResult for the onDone settle no longer leaves the service dropped live but unrecorded, which made Recover run the ServiceFn again. - durable: an effect handler with a Store that isn't a DispatchStore is refused with ErrEffectDispatch by Start, Recover and the stateless Runner.Fire, before any Store read or write or handler call. On main it silently skipped every effect. The no-DispatchStore no-op branches in dispatch are gone; no handler still dispatches nothing. Each change has a test that fails without it, checked by reverting it. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…ol for a stale settle - A Tick's timer transition no longer discards a parked result, live or on replay. Tick hands the transition's effects only to the scheduler, so the ServiceRunner keeps the service in flight; discarding the parked result made the next RunService run the ServiceFn again. Tick still forgets settled services, so Recover starts the service of a state a timer re-entered, as TestRecover_StartsAServiceThatNeverSettled pins. #236 stops the service and then forgets both. - A Recover from a checkpoint taken after the timer left the state still finds the service neither in flight nor parked, while live and the journal's replay keep it in flight: a skipped test pins the agreement #236 brings. - RunService's doc says the bool means the service was in flight and ran, not that it settled: a settle that finds its activation stopped or armed again returns true with the zero result and records nothing. The doc no longer says a timer that leaves the state discards the parked result. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this change does
Domain effects now reach the handler from
Start,TickandRunService, not onlyFire.TickandRunServicerecord their step the wayFiredoes: append under the step lock, then dispatch the effects outside it, before any checkpoint.Recoverretries the effect, and fails withErrEffectDispatchwhile the handler still does. This changes behavior: aFireafter a failed dispatch used to be accepted.Startencodes the baseline, applies the initial entry's effects, then writes the baseline, and applies none if the entry failed. Those effects are at-least-once: a handler failure, a failed baseline write or overlappingStartcalls re-apply them. Nothing can re-derive them after the baseline. Tests pin this contract; an exactly-once design was tried and needs a new public effect-registry option to rebuild the effect onRecover.ErrEffectDispatchfromStartandRecover, as it already did from a live step.durable.EffectInstance(ctx)gives a handler the instance, because effect ids repeat across instances (durable: defects found in the platform-redesign review #214 question 5). The proposed answer was to add the instance to the effect id. This takes the issue's other option and passes the instance to the handler, so the dispatched set needs no migration.state.ServiceRunner.WithResultTransform(func(event E, result any, err error) (out any, outErr error, abort error)), an additive change to the frozenstatemodule. It runs on every settle (Tick,SettleDone,SettleError) and gets the event that will carry the outcome. A non-nilabortfires no event and leaves the service in flight; the settle methods return it inFireResult.Errwithfalse. An unbound service still settles through onError.durableuses it to decode a service result throughWithEventDataDecoder, keyed by the onDone event, the same way live and on replay. With a decoder registered,Assignsees the registered type; without one, the JSON-generic form. A decoder that fails, or decodes to a different type than the service returned, parks the result:RunServicereturnsErrEventDataEncoding, the Handle stays healthy, and noAssignruns. The raw result is journaled, and kept in the checkpoint (an additiveparkedServicesfield), so the ServiceFn never runs again. Fix the decoder and callRunServiceagain to settle it. Leaving the state discards it.KindedEffectwhose payload can't marshal no longer fails a step: it's recorded by id and kind, with no payload, and the error underMeta["crucible.marshalError"].Recoverre-derives its value, so dispatch is unaffected. This relaxesFire, which failed in this case on main.Please check these:
Recoverfails while the handler is down. A handler that always refuses an effect wedges the instance. This needs a ruling.WithEffectHandleron a Store that isn't aDispatchStoreis now refused, instead of silently applying no effects.NewRunnercan't return an error, soStart,Recoverand the statelessFirerefuse before any Store read, write or handler call;StateAtandStepsdon't dispatch, so they don't. The error wrapsErrEffectDispatch. On main such a Runner skipped Fire's effects andRecover's re-send; with this PR's dispatch it would have skipped Start, Tick and RunService too.MemStoreandFileStoreboth implementDispatchStore, so only a custom Store withoutDispatchedandMarkDispatchedis affected. I'd suggest refusing, but it's your call: the alternatives are dispatching at-least-once with no dedupe on such Stores, or keeping the documented silent skip. One more call inside it: wrappingErrEffectDispatchmeans a caller that retriesRecoveron that error also retries this configuration error; a plain error or a newErrNoDispatchStoresentinel would avoid that.state, for your ratification:ServiceRunner.Tickreturns an empty slice when the service was stopped, or stopped and armed again, while its ServiceFn ran, as it does for an id not in flight. A non-empty slice now always means the settle fired its event or was aborted. This PR first returned one zeroFireResultfor that case, whichdurablehad to read from the zero value, so a middleware returningFireResult{}for a real settle was misread and the service ran again afterRecover. Neither form has been released; the signature doesn't change.Fire, aren't dispatched.ActorSystemdrops those results, so this needs astatechange. durable: defects found in the platform-redesign review #214 lists it.Assignsees a struct as a map and an integer as afloat64that loses precision above 2^53, live as on replay. Errors are message-only either way, soerrors.Isdoesn't match in onError. A decoder added or changed after results were recorded changes what replay produces: register it before recording.errors.Is.Recoverre-dispatches only Records that name their effects, so an older journal's uncompactedTickand service steps don't send effects they never sent.journalFormatis now 2. A service result from a format 0 or 1 journal replays as generic JSON, with no decoder, as the older version did. The journal records the exact JSON the live settle decoded.Recover, the statelessFire,StateAtandSteps, before anything replays, with an error naming both formats. Versions already built don't have this check, so rolling back past this version isn't protected.MarkDispatchedfails the Handle, andRecoverapplies that effect again.Why
Effects from entry, timer and service transitions never reached the handler, and a failed effect could be compacted away and never retried.
Relates to: #214
Evidence
On
GOTOOLCHAIN=go1.27.4, with-race, indurableunless the row says otherwise:go test ./...instateresult_transform_test.go(rewrite, abort, unbound service, panic)go test ./...TestRunner_EffectDispatch_*,effect_recover_test.go,service_outcome_test.go,result_decoding_scenarios_test.go(one skip:errors.Ison a service error, message-only by design) andstart_effects_contract_test.gogolangci-lintv2.15.0 indurableandstateChecklist
git commit -s) per the DCOtype: subject)mage checkpasses locally (lint + race tests + govulncheck): not run as a whole