Skip to content

fix(durable): dispatch effects from Start, Tick and RunService - #235

Draft
reka18 wants to merge 13 commits into
mainfrom
fix/durable-effect-dispatch
Draft

reka18 wants to merge 13 commits into
mainfrom
fix/durable-effect-dispatch

Conversation

@reka18

@reka18 reka18 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

What this change does

Domain effects now reach the handler from Start, Tick and RunService, not only Fire.

  • Tick and RunService record their step the way Fire does: append under the step lock, then dispatch the effects outside it, before any checkpoint.
  • A failed dispatch fails the Handle before its checkpoint. Recover retries the effect, and fails with ErrEffectDispatch while the handler still does. This changes behavior: a Fire after a failed dispatch used to be accepted.
  • Start encodes the baseline, applies the initial entry's effects, then writes the baseline, and applies none if the entry failed. Those effects are at-least-once: a handler failure, a failed baseline write or overlapping Start calls re-apply them. Nothing can re-derive them after the baseline. Tests pin this contract; an exactly-once design was tried and needs a new public effect-registry option to rebuild the effect on Recover.
  • A panicking effect handler now returns ErrEffectDispatch from Start and Recover, as it already did from a live step.
  • New durable.EffectInstance(ctx) gives a handler the instance, because effect ids repeat across instances (durable: defects found in the platform-redesign review #214 question 5). The proposed answer was to add the instance to the effect id. This takes the issue's other option and passes the instance to the handler, so the dispatched set needs no migration.
  • New state.ServiceRunner.WithResultTransform(func(event E, result any, err error) (out any, outErr error, abort error)), an additive change to the frozen state module. It runs on every settle (Tick, SettleDone, SettleError) and gets the event that will carry the outcome. A non-nil abort fires no event and leaves the service in flight; the settle methods return it in FireResult.Err with false. An unbound service still settles through onError.
  • durable uses it to decode a service result through WithEventDataDecoder, keyed by the onDone event, the same way live and on replay. With a decoder registered, Assign sees the registered type; without one, the JSON-generic form. A decoder that fails, or decodes to a different type than the service returned, parks the result: RunService returns ErrEventDataEncoding, the Handle stays healthy, and no Assign runs. The raw result is journaled, and kept in the checkpoint (an additive parkedServices field), so the ServiceFn never runs again. Fix the decoder and call RunService again to settle it. Leaving the state discards it.
  • A KindedEffect whose payload can't marshal no longer fails a step: it's recorded by id and kind, with no payload, and the error under Meta["crucible.marshalError"]. Recover re-derives its value, so dispatch is unaffected. This relaxes Fire, which failed in this case on main.

Please check these:

  • Departs from the proposed answer to durable: defects found in the platform-redesign review #214 question 4 ("Recover now, redispatch later"): the Handle stops, and Recover fails while the handler is down. A handler that always refuses an effect wedges the instance. This needs a ruling.
  • Behavior change for your ratification (durable: defects found in the platform-redesign review #214 question 3): a Runner with WithEffectHandler on a Store that isn't a DispatchStore is now refused, instead of silently applying no effects. NewRunner can't return an error, so Start, Recover and the stateless Fire refuse before any Store read, write or handler call; StateAt and Steps don't dispatch, so they don't. The error wraps ErrEffectDispatch. On main such a Runner skipped Fire's effects and Recover's re-send; with this PR's dispatch it would have skipped Start, Tick and RunService too. MemStore and FileStore both implement DispatchStore, so only a custom Store without Dispatched and MarkDispatched is affected. I'd suggest refusing, but it's your call: the alternatives are dispatching at-least-once with no dedupe on such Stores, or keeping the documented silent skip. One more call inside it: wrapping ErrEffectDispatch means a caller that retries Recover on that error also retries this configuration error; a plain error or a new ErrNoDispatchStore sentinel would avoid that.
  • Behavior change in state, for your ratification: ServiceRunner.Tick returns an empty slice when the service was stopped, or stopped and armed again, while its ServiceFn ran, as it does for an id not in flight. A non-empty slice now always means the settle fired its event or was aborted. This PR first returned one zero FireResult for that case, which durable had to read from the zero value, so a middleware returning FireResult{} for a real settle was misread and the service ran again after Recover. Neither form has been released; the signature doesn't change.
  • Not fixed: effects from a parent transition an actor drives, including during Fire, aren't dispatched. ActorSystem drops those results, so this needs a state change. durable: defects found in the platform-redesign review #214 lists it.
  • With no decoder registered, a service's Assign sees a struct as a map and an integer as a float64 that loses precision above 2^53, live as on replay. Errors are message-only either way, so errors.Is doesn't match in onError. A decoder added or changed after results were recorded changes what replay produces: register it before recording.
  • A handler's error now matches through errors.Is.
  • Recover re-dispatches only Records that name their effects, so an older journal's uncompacted Tick and service steps don't send effects they never sent.
  • journalFormat is now 2. A service result from a format 0 or 1 journal replays as generic JSON, with no decoder, as the older version did. The journal records the exact JSON the live settle decoded.
  • A journal a newer version wrote is refused by Recover, the stateless Fire, StateAt and Steps, before anything replays, with an error naming both formats. Versions already built don't have this check, so rolling back past this version isn't protected.
  • A failed MarkDispatched fails the Handle, and Recover applies that effect again.

Why

Effects from entry, timer and service transitions never reached the handler, and a failed effect could be compacted away and never retried.

Relates to: #214

Evidence

On GOTOOLCHAIN=go1.27.4, with -race, in durable unless the row says otherwise:

Command Result
go test ./... in state Pass, including result_transform_test.go (rewrite, abort, unbound service, panic)
go test ./... Pass, including TestRunner_EffectDispatch_*, effect_recover_test.go, service_outcome_test.go, result_decoding_scenarios_test.go (one skip: errors.Is on a service error, message-only by design) and start_effects_contract_test.go
golangci-lint v2.15.0 in durable and state 0 issues

Checklist

  • Commits are signed off (git commit -s) per the DCO
  • Conventional commit messages (type: subject)
  • mage check passes locally (lint + race tests + govulncheck): not run as a whole
  • Tests added/updated for the change
  • Public API changes documented (godoc) and, if applicable, noted for the changelog

@reka18 reka18 added not-ready On every new PR until all review dimensions are confirmed wait:correctness ACR pending: correctness wait:regression-safety ACR pending: regression-safety wait:concurrency-discipline ACR pending: concurrency-discipline wait:replay-determinism ACR pending: replay-determinism wait:api-compatibility ACR pending: api-compatibility wait:missed-context ACR pending: missed-context wait:human-readability ACR pending: human-readability labels Nov 5, 2026
@reka18
reka18 force-pushed the fix/durable-effect-dispatch branch from ebb5863 to ef9c0a8 Compare November 5, 2026 16:38
@reka18
reka18 force-pushed the fix/durable-effect-dispatch branch from ef9c0a8 to d46adf2 Compare November 5, 2026 16:40
@reka18
reka18 force-pushed the fix/durable-effect-dispatch branch from d46adf2 to cc4d8cc Compare November 5, 2026 16:53
@reka18
reka18 force-pushed the fix/durable-effect-dispatch branch from cc4d8cc to d657891 Compare November 5, 2026 17:09
@reka18
reka18 force-pushed the fix/durable-effect-dispatch branch from d657891 to 3e41947 Compare November 5, 2026 17:14
@reka18
reka18 force-pushed the fix/durable-effect-dispatch branch from 3e41947 to 3edd2ee Compare November 5, 2026 17:22
@reka18
reka18 force-pushed the fix/durable-effect-dispatch branch from 3edd2ee to aae543d Compare November 5, 2026 17:33
@reka18
reka18 added this pull request to stack #238 November 5, 2026 17:34
@reka18
reka18 force-pushed the fix/durable-effect-dispatch branch from aae543d to 50bbc98 Compare November 5, 2026 17:42
@reka18
reka18 force-pushed the fix/durable-effect-dispatch branch from 50bbc98 to a14c4ed Compare November 5, 2026 18:04
@reka18
reka18 force-pushed the fix/durable-effect-dispatch branch from a14c4ed to 90bcbfb Compare November 5, 2026 18:20
@reka18 reka18 added confirmed:concurrency-discipline ACR clean: concurrency-discipline confirmed:replay-determinism ACR clean: replay-determinism confirmed:api-compatibility ACR clean: api-compatibility confirmed:missed-context ACR clean: missed-context confirmed:human-readability ACR clean: human-readability and removed wait:concurrency-discipline ACR pending: concurrency-discipline wait:replay-determinism ACR pending: replay-determinism wait:api-compatibility ACR pending: api-compatibility wait:missed-context ACR pending: missed-context wait:human-readability ACR pending: human-readability labels Nov 5, 2026
@reka18

reka18 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor Author

Review results. One reviewer per dimension, each reading the others' findings. All seven dimensions have zero blockers on the latest round.

Fixed along the way: EffectInstance replaced advice to dedupe on an id that repeats across instances. The docs now state when an effect can apply twice, and that a service result reaches Assign JSON-decoded, with an integer losing precision above 2^53. examples/fooddelivery accepts the decoded amount. The four effect-dispatch tests moved here from #213.

For a maintainer: questions 3 and 4 on #214 still need a ruling. The description says where this PR departs from the proposed answers.

Only Fire dispatched domain effects. Tick and RunService now record a step
through the same helper as Fire, then dispatch its effects as Fire does, after
the append and before any checkpoint. A failed dispatch fails the
Handle before its checkpoint, so a later checkpoint can't compact the effect
away, and Recover retries it. Start applies the initial entry's effects before
the baseline, so a handler failure leaves no instance.

EffectInstance gives a handler its instance id, and state.ServiceRunner gains
WithResultTransform so a service's result reaches the machine in the form a replay
supplies.

Effects from a parent transition an actor drives are still not dispatched.

Relates to #214.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…s a replay can read

Review follow-ups for the effect dispatch change, each pinned by a test.

- One helper, applyEffect, calls the effect handler for Start, a step and
  Recover, and turns a handler panic into ErrEffectDispatch. Start and Recover
  used to let the panic out, for a Fire's effects as well as a tick's or a
  service settle's.
- A service result whose JSON form won't decode, such as a number beyond
  float64, now fails RunService with ErrHandleFailed and records nothing, as a
  result that won't encode does. It used to be journaled, and every Recover
  then failed to decode it.
- A service error whose Error method panics settles with "service error
  panicked: ...", its invalid UTF-8 replaced, so live and replay see the same
  message. encodeEventData and replayableOutcome share normalizedErrorMessage,
  which drops replayableOutcome's dead fallbacks.
- Recover re-dispatches a step's effects only when its Record names them, so it
  doesn't send effects from an older version's Tick or service steps, which
  never dispatched them.
- state: a WithResultTransform that returns no error for an unbound service no
  longer turns it into a success; the service still settles through onError.
- Tests that Recover re-dispatches a tick's, a multi-timer tick's and a service
  settle's unmarked effects once, under the ids the live run stamped.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…in Start's at-least-once effects

A service result now decodes through the Runner's EventDataDecoder for the
invocation's onDone event, live and on replay, as a WithEventData payload does.
A decoder for that event hands Assign its type; without one the result arrives
in JSON's generic form. A service error stays message-only.

- state: ServiceRunner.WithResultTransform's function takes the event that will
  carry the outcome, func(event E, result any, err error) (any, error), and runs
  on every settle: Tick, SettleDone and SettleError. It runs before any event
  fires. An UnboundRefError still settles through onError when the transform
  returns no error.
- A decoder that fails on a result, or returns a type other than the service
  returned, fails RunService with ErrHandleFailed wrapping ErrEventDataEncoding
  before any onDone or onError Assign runs, and records nothing. Recover and
  StateAt fail with ErrEventDataEncoding on a recorded result their decoder can't
  decode, also running no Assign. This also covers a result whose JSON form won't
  decode, such as a number beyond float64.
- The docs say to register the decoder before results are recorded: replay
  decodes the journal with the recovering Runner's decoder.
- Tests pin the decoding scenarios, that a decode failure runs no Assign, and
  the documented hazard of a decoder added after recording.
- Tests pin Start's initial effects as at-least-once: applied before the
  baseline write and never marked, so a handler failure, a failed baseline
  write or overlapping Starts re-apply them, and Recover never applies them.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…icking

ServiceRunner.WithResultTransform's function now returns an explicit abort:
func(event E, result any, err error) (out any, outErr error, abort error).

- A non-nil abort stops the settle: no event fires, so no Assign, middleware,
  inspector or history entry. SettleDone and SettleError return false with the
  abort in FireResult.Err; Tick returns that FireResult as its one element.
  LastResult and LastError keep the previous settle.
- The transform runs before the service is dropped, so an aborted settle, or a
  transform that panics, leaves the service in flight. A panic in the transform
  goes on out of the settle, as any host panic does.
- durable's transform returns an abort for a result that won't decode for its
  onDone event, in place of the resultDecodeFailure panic. RunService and replay
  check the returned abort; tickService, settleRecorded and their recover() are
  gone. Live, the step still fails the Handle and records nothing, and Recover has
  the service in flight again; replay still fails with ErrEventDataEncoding.
- TestWithResultTransform_AbortFiresNoEvent replaces the panic test, covering Tick,
  SettleDone and SettleError.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
… results

Each finding gets a test that fails if it's real, committed whether it proved
or disproved the finding.

- state: a settle can't settle a later activation of its service. Each
  activation carries a generation; a SettleDone or SettleError whose transform
  ran, or a Tick whose ServiceFn ran, while the service was stopped and armed
  again under the same invoke id now returns false and leaves the new
  activation in flight.
- durable: a DispatchStore whose Dispatched or MarkDispatched panics fails
  Recover and the stateless Runner.Fire with ErrEffectDispatch, as a step
  already did. The guard moves into the shared dispatch path, which
  applyStartEffects now shares its handler and DispatchStore checks with.
- durable: a decoder may return json.Number, JSON's generic form under
  UseNumber, for any service result. The service and Fire paths share one type
  check; a Fire payload stays strict.
- durable: DeliverToActor records through recordStep and finishStep;
  persistStep is gone.
- Pins: an undecodable service result reruns the ServiceFn on every Recover;
  an unmarshalable KindedEffect fails Tick and RunService with no handler
  wired; Recover loads the dispatched set once per effect Record, as main did
  for Fire Records; the journal holds the decoded result's encoding, which
  differs from the first encoding for a generic struct or an int64 above 2^53.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…ct that won't marshal by id and kind

- durable: a service result the decoder for its onDone event refuses, or decodes
  as another type than the service returned, is parked. RunService returns
  ErrEventDataEncoding, not ErrHandleFailed, before any Assign runs, and records
  the result with its Go type in a Record that settles nothing. The Handle stays
  healthy and the service stays in flight, but its ServiceFn never runs again:
  replay parks the result again, and a RunService once the decoder decodes it
  settles the parked result, held to the type the service returned. The park
  step offers its checkpoint as any step does, and the checkpoint envelope
  carries parked results in a new parkedServices field, so compaction can't
  lose one. Leaving the state that declares the service discards the parked
  result; entering it again runs the ServiceFn for the new activation. StateAt
  replays a parked Record as a step that changed nothing. state's
  ServiceRunner is unchanged: a parked service is in flight there.
- durable: a KindedEffect whose payload won't marshal is recorded by id and
  kind with no payload, as an effect that isn't a KindedEffect is, with the
  marshal error's message in Meta under crucible.marshalError. No step fails
  on it, Fire included, and it dispatches as any effect does.
- The pins for both now assert the new behavior: the ServiceFn runs once, and
  every kind of step records the effect.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
… format

Each finding gets a test that fails if it's real, committed whether it proved or
disproved the finding.

- durable: a ctx that ended during the ServiceFn parks nothing. A result the
  decoder refuses fails the Handle and records nothing, as a result that decodes
  does, and the service runs again after Recover.
- durable: the journal records the JSON the live settle decoded, not the decoded
  value encoded again, so replay decodes the same bytes even for a decoder type
  whose MarshalJSON and UnmarshalJSON aren't inverses, or whose decoded value
  won't encode again. The second marshal is gone.
- durable: journalFormat is 2. A service result in a Record of format 0 or 1, which
  an older version wrote when live saw the ServiceFn's own value, replays in JSON's
  generic form through no decoder, as that version replayed it, so a decoder added
  for its onDone event can't fail Recover or StateAt on it.
- durable: a parked result records its type with its import path, so a decoder for
  another package's type of the same short name can't settle it.
- durable: a result with no JSON form aborts the settle, so no Assign runs, and
  fails the Handle as before.
- durable: a KindedEffect whose MarshalJSON panics is recorded by id and kind with
  the panic's message under crucible.marshalError, rather than panicking out of a
  step the kernel applied. A Kind that panics still fails the Handle; a test pins it.
- durable: replay decodes each service payload once, and typeName runs only for a
  park. parkedServices dedupes with a sorted set, as endedServices does.
- docs: the Handle that parked a result never sees a fixed decoder; every doc site
  now says to Recover with the fixed decoder, then call RunService.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
Recover, the stateless Runner.Fire, StateAt and Steps now refuse a journal
whose checkpoint or any Record records a journal format above this
version's, before they restore or replay anything. The check covers a
checkpoint with an empty tail, so each later bump of journalFormat
protects a rollback. Formats 0, 1 and 2 still load.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…ixed versions and stale settles

Each finding has a test that fails if it is real, kept whether it proved or
disproved it, and each fix was checked by reverting it.

- Recover sends what replaying a step produces by the Record's format, not its
  Effects envelopes: a Fire's always, a tick's or a service settle's from
  format 2. A Store that drops Record.Effects no longer loses an effect a crash
  left unsent; an older version's tick and service effects still aren't sent.
- Recover loads the dispatched set once, not once per Record with effects:
  1000 tick Records go from about 22 ms to 2 ms on MemStore, 25 ms to 4 ms on
  FileStore.
- A DispatchStore error in Dispatched or MarkDispatched wraps ErrEffectDispatch,
  as a panic there already did; the ErrEffectDispatch doc now covers the
  dispatch tracking.
- The Record that settles a parked result records its type, and replay holds
  the decoder to it, as the settle did.
- A parked result's type name keeps the import path inside struct literal, func,
  chan and interface types, with struct tags and unexported fields' packages.
- A Record of a lower journal format after a newer checkpoint is reported as an
  older version appending to a newer journal; a Record with no format still
  blames the Store. The CHANGELOG says not to run mixed versions on one instance.
- A settle that finds its activation stopped or armed again records nothing.
  The step lock keeps durable's own steps from causing it; an internal test
  drives the ServiceRunner directly.
- The JSON a settle decoded is cleared once the step, or Recover's replay, has
  used it.
- Settled services and parked results are updated together through one helper
  on the Handle and one on the replay mirror.
- Pins: a moved or renamed type leaves a parked result parked under a typed
  decoder, and JSON's generic form still settles it; a ServiceFn can't change
  its own activation through the Handle.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…'s type, reread the dispatched set before unmarked effects

Each finding has a test that fails if it is real, kept whether it proved or
disproved it, and each fix was checked by reverting it.

- A Record of a lower journal format than any before it, the checkpoint's or an
  earlier Record's, is refused as an older version appending to a newer journal,
  not only one below the checkpoint's. Recover and StateAt check every Record's
  format before they replay any, so a refused journal sends no effect.
- Every successful service settle records the Go type the service returned, and
  replay holds the decoder to it, as for a settled parked result. JSON's generic
  form and nil are always accepted, and a Record of format 0 or 1 still replays
  with no check. Changing the type a decoder returns for an event with recorded
  results now fails Recover unless it returns the generic form; the RunService
  doc, package doc and CHANGELOG say so.
- Recover reads the dispatched set again before sending a step's effects that its
  copy doesn't show marked, so a mark a stale owner makes during a handoff is seen,
  as the per-step load did. A step whose effects are all marked costs no read:
  1000 landed tick effects still take one read, and about 2 ms on MemStore.
- The tick-tail benchmark also measures a tail with no effect landed, and its run
  line names it correctly.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…n effect handler on a Store without dispatch tracking

- state: ServiceRunner.Tick returns an empty slice when the activation its
  ServiceFn ran for was stopped, or stopped and armed again, before the settle,
  as for an id not in flight. An abort still returns its one-element slice with
  the abort in Err. A non-empty slice now always means the settle fired or was
  aborted, whatever FireResult the Fire returned.
- durable: serviceStep reads a stale settle from Tick's empty result, and
  settledNothing, which guessed it from a zero FireResult, is gone. A middleware
  that returns a zero FireResult for the onDone settle no longer leaves the
  service dropped live but unrecorded, which made Recover run the ServiceFn
  again.
- durable: an effect handler with a Store that isn't a DispatchStore is refused
  with ErrEffectDispatch by Start, Recover and the stateless Runner.Fire, before
  any Store read or write or handler call. On main it silently skipped every
  effect. The no-DispatchStore no-op branches in dispatch are gone; no handler
  still dispatches nothing.

Each change has a test that fails without it, checked by reverting it.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…ol for a stale settle

- A Tick's timer transition no longer discards a parked result, live or on
  replay. Tick hands the transition's effects only to the scheduler, so the
  ServiceRunner keeps the service in flight; discarding the parked result made
  the next RunService run the ServiceFn again. Tick still forgets settled
  services, so Recover starts the service of a state a timer re-entered, as
  TestRecover_StartsAServiceThatNeverSettled pins. #236 stops the service and
  then forgets both.
- A Recover from a checkpoint taken after the timer left the state still finds
  the service neither in flight nor parked, while live and the journal's replay
  keep it in flight: a skipped test pins the agreement #236 brings.
- RunService's doc says the bool means the service was in flight and ran, not
  that it settled: a settle that finds its activation stopped or armed again
  returns true with the zero result and records nothing. The doc no longer says
  a timer that leaves the state discards the parked result.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

confirmed:concurrency-discipline ACR clean: concurrency-discipline confirmed:missed-context ACR clean: missed-context confirmed:regression-safety ACR clean: regression-safety not-ready On every new PR until all review dimensions are confirmed wait:api-compatibility ACR pending: api-compatibility wait:correctness ACR pending: correctness wait:human-readability ACR pending: human-readability wait:replay-determinism ACR pending: replay-determinism

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant