Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 25 additions & 10 deletions docs/src/content/docs/durable/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,12 @@ implement the `Store` interface out of tree.
- **Deterministic replay.** A recovered instance is byte-identical to one that
never crashed, because recovery replays the recorded driving events and
nondeterministic results rather than re-executing their sources.
- **Exactly-once effects.** A domain effect is applied exactly once over the
- **Exactly-once effects.** A domain effect from a step (not an initial entry, which is at-least-once, or a parent transition an actor drives, which is not dispatched) is applied exactly once over the
instance's lifetime (the live run plus any number of recoveries), even though
the replay loop is at-least-once. Each effect carries a deterministic
`EffectID` and is deduplicated through the `Store`'s dispatch set.
the replay loop is at-least-once. The exception is a crash or failed mark after the
handler returned, or a handoff between owners, where it can apply twice, so keep
handlers idempotent on `EffectInstance` plus the `EffectID`. Each effect carries a
deterministic `EffectID` and is deduplicated through the `Store`'s dispatch set.
- **Durability across restart.** Every `Fire` step is write-ahead appended to the
`Store` before it is acknowledged, so a crash after a successful `Fire` never
loses the step. Periodic checkpoints bound the tail that recovery replays.
Expand Down Expand Up @@ -72,9 +74,9 @@ the first time, and replayed verbatim on recovery:
| Seam | Wire with | Recorded as |
| --- | --- | --- |
| **Clock** (timers) | `WithRunnerClock` | `JournalClockRead`, replayed so timers fire at the same recorded instants, wall-clock-independent; armed deadlines survive checkpoint compaction |
| **Invoked services** | `WithServiceRegistry` + `Handle.RunService` | `JournalServiceResult`; the service runs once, then recovery replays its result through the kernel's settle seam |
| **Child-machine actors** | `WithActorPalette` + `Handle.DeliverToActor` | `JournalActorMessage`; the behavior runs once, then recovery re-fires the recorded parent transition |
| **Domain effects** | `WithEffectHandler` | dispatch set, applied exactly once via deterministic `EffectID` dedup |
| **Invoked services** | `WithServiceRegistry` + `Handle.RunService` | `JournalServiceResult`; a recorded result replays through the kernel's settle seam without re-running the service, and an unrecorded one (a context canceled during the service, a failed `Append`, a crash) runs again when the host calls `RunService` |
| **Child-machine actors** | `WithActorPalette` + `Handle.DeliverToActor` | `JournalActorMessage`; recovery re-fires a recorded parent transition; a settled actor doesn't re-run, and a running one restarts from its initial state on any `Recover` |
| **Domain effects** | `WithEffectHandler` | dispatch set, a step's effects applied exactly once via deterministic `EffectID` dedup, with the exceptions under Exactly-once effects |

Use `WithCheckpointEvery(n)` to tune how often a full snapshot is written: a
shorter interval bounds recovery replay, a longer one cuts checkpoint cost.
Expand Down Expand Up @@ -125,14 +127,27 @@ Time-travel needs the full record history through the target step. A `Store` opt
in by implementing `HistoryStore` (the in-tree `MemStore` does so under
`WithHistory`); `StateAt` otherwise falls back to the latest checkpoint plus tail.

## Handles across goroutines

`Fire`, `Tick`, `RunService` and `DeliverToActor` are safe to call from different
goroutines: each holds the Handle's lock for the whole step. A host callback must not
call back into its Handle. `Instance()` is outside the lock.

A step that may not have been recorded, a failed effect dispatch, or a `Tick`, `RunService`
or `DeliverToActor` whose context ended partway, fails the Handle with `ErrHandleFailed`.
`Recover` a new Handle with the same options to continue, unless the cause is `ErrFenced`:
then a newer owner holds the instance, so stop. `Recover` retries an unapplied effect, and
returns `ErrEffectDispatch` while the handler still fails.

## A note on serialized payloads

Events, service done-data, actor done-data, and actor messages are recorded as their
JSON form. A parent reducer that type-asserts a non-JSON Go type from `AssignCtx.Event`
observes the JSON-decoded shape on a replayed `onDone`. A `WithEventData` payload is
JSON-decoded live as well as on replay, so a reducer gets the same value both ways: a
struct is a map and an integer a `float64`. An error stays an error with the same
message, with any invalid UTF-8 replaced. `Handle.Fire` refuses a payload it can't
observes the JSON-decoded shape of actor done-data on a replayed `onDone`. A
`WithEventData` payload and a service's result are JSON-decoded live as well as on replay,
so a reducer gets the same value both ways: a struct is a map and an integer a `float64`,
which loses precision above 2^53. An error stays an error with the same message, with any
invalid UTF-8 replaced, and `errors.Is` no longer matches it. `Handle.Fire` refuses a payload it can't
record, or one over `WithMaxEventDataBytes` (8 MiB by default), with
`ErrEventDataEncoding`, and nothing applies. A
typed-codec option to carry the concrete Go value across the journal boundary is
Expand Down
62 changes: 62 additions & 0 deletions durable/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ and this module adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.

### Added

- **`EffectInstance`** reads the instance from the ctx an `EffectHandler` receives, because
effect ids repeat across instances.
- **Fencing.** `WithFence` on `Store.Append` and `WithCheckpointFence` on
`Store.Checkpoint` carry a caller-issued token, and a Store refuses a token lower
than the highest it has seen for the instance with `ErrFenced`, writing nothing.
Expand All @@ -25,9 +27,37 @@ and this module adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.
a refused payload, and `ErrRecordTooLarge` a Record too large for a `FileStore` line.
`Record.EventData` and `Record.EventDataError` hold the payload.

- **`ErrHandleFailed`** reports a Handle out of step with its journal.

### Changed

- Requires Go 1.27.3.
- **A failed effect dispatch fails the Handle.** The call returns `ErrHandleFailed`
wrapping `ErrEffectDispatch`, before any checkpoint, so the effect can't be compacted
away. `Recover` retries it, and fails with `ErrEffectDispatch` while the handler still
does. A `Fire` after a failed dispatch used to be accepted. A failed `MarkDispatched`
fails the Handle too, and `Recover` applies that effect again. The handler's error now
matches through `errors.Is`.
- **A `KindedEffect` that can't marshal fails `Tick` and `RunService`,** as it already
failed `Fire`, even with no handler wired.
- **A service result reaches `Assign` decoded, live as well as on replay,** so a retried
effect can't differ from the first attempt: a struct arrives as a map, an integer as a
`float64`, which loses precision above 2^53, and an error as an error with the same
message, any invalid UTF-8 replaced, which `errors.Is` no longer matches. An `Error`
method that panics settles with a message saying so. The journal records this decoded
value.
- **A Handle's steps are serialized across goroutines.** `Fire`, `Tick`, `RunService` and
`DeliverToActor` hold a lock for the whole step, so a host's timer loop no longer loses
acknowledged steps. A host callback must not call back into its Handle, or it
deadlocks. `Instance()` isn't covered: read it only while no step is running. A `Tick`,
`RunService` or `DeliverToActor` whose context ended while it waited returns that error
and changes nothing.
- **A step that may not have been recorded fails the Handle.** That call and every later
one return `ErrHandleFailed`, wrapping the cause. `Recover` a new Handle with the same
options to continue. A service whose result wasn't recorded is in flight again, so
`RunService` runs it a second time. An `InvokeActor` actor restarts from its initial
state, and a `Spawn` actor doesn't return. If the cause is `ErrFenced`, a newer owner
holds the instance: stop.
- **A `WithEventData` payload reaches `Assign` decoded, live as well as on replay.** A
struct arrives as a map and an integer as a `float64`, which loses precision above
2^53, so an assertion to the original
Expand Down Expand Up @@ -59,6 +89,38 @@ and this module adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.

### Fixed

- **A failed checkpoint no longer makes the next step conflict.** `Tick`, `RunService`
and `DeliverToActor` now advance the next step as soon as the append lands.
- **Timers survive recovery.** `Start` now arms the initial state's timers, which it never
did, so a new instance starting in a timed state fires a timer it didn't before. It
carries them in the baseline; an instance started earlier keeps its initial timer
unarmed. Recovery re-arms checkpointed timers before the tail replays, so one fired in
the tail isn't missed, and a timer armed by a service result now replays. `StateAt`
re-arms baseline timers too, so its result changes when one fires in the window, and it
fails if a timer's event won't decode with the options passed.
- **A tick that fires nothing records nothing.** It used to append a Record every time,
so a timer loop grew the journal while a timer was pending. An idle `Tick` doesn't touch
the Store, so it can't report `ErrFenced` or a Store failure: a stale owner finds out on
its next real step. Replay still reads older journals, whose idle Records and misplaced
settle clock reads it aligns on. Two kinds of older journal can't be repaired, and can
recover differently or fail `Recover`: an instance that continued after an older
recovery missed a fired timer, and a checkpoint taken after a tick that followed a settle
or whose clock moved mid-tick, which can hold a wrong timer table.
- **Effects reach the handler from `Start`, `Tick` and `RunService`.** They were only
dispatched from `Fire`. `Start` applies the initial entry's effects after encoding the
baseline and before writing it, and only if the entry succeeded. Those effects are
at-least-once: a handler failure, a failed baseline write or overlapping `Start` calls
re-apply them, so key any dedupe on `EffectInstance` plus the effect id. Effects from a
parent transition an actor drives, including during `Fire`, are still not dispatched.
The first `Recover` of an older journal applies effects from uncompacted `Tick` and
service steps.
- **A step whose context ended partway isn't journaled.** A canceled `RunService`, `Tick`
or `DeliverToActor` used to record a result the live instance never applied, which
recovery then committed, or for an actor, failed on. Each now returns `ErrHandleFailed`
wrapping the error and records nothing. After `Recover` the service stays in flight and
must tolerate running again, the timer is re-armed, and the actor restarts. `Recover`
and `StateAt` now fail on a canceled context, rather than returning state short of the
journal.
- **The next step survives a checkpoint that compacts tick barriers.** A checkpoint
now records the step that follows it, in a `nextStep` field of its envelope. Recovery
used the Trace count, which runs low past a tick barrier, so the next Fire could reuse a
Expand Down
35 changes: 25 additions & 10 deletions durable/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,12 @@ backends implement the `Store` interface out of tree.
- **Deterministic replay**: a recovered instance is byte-identical to one that
never crashed, because recovery replays the recorded driving events and
nondeterministic results rather than re-executing their sources.
- **Exactly-once effects**: a domain effect is applied exactly once over the
- **Exactly-once effects**: a domain effect from a step (not an initial entry, which is at-least-once, or a parent transition an actor drives, which is not dispatched) is applied exactly once over the
instance's lifetime (the live run plus any number of recoveries), even though
the replay loop is at-least-once. Each effect carries a deterministic
`EffectID` and is deduplicated through the `Store`'s dispatch set.
the replay loop is at-least-once. The exception is a crash or failed mark after the
handler returned, or a handoff between owners, where it can apply twice, so keep
handlers idempotent on `EffectInstance` plus the `EffectID`. Each effect carries a
deterministic `EffectID` and is deduplicated through the `Store`'s dispatch set.
- **Durability across restart**: every `Fire` step is write-ahead appended to
the `Store` before it is acknowledged, so a crash after a successful `Fire`
never loses the step. Periodic checkpoints bound the tail recovery replays.
Expand Down Expand Up @@ -82,9 +84,9 @@ the first time, and replayed verbatim on recovery:
| Seam | Wire with | Recorded as |
| --- | --- | --- |
| **Clock** (timers) | `WithRunnerClock` | `JournalClockRead`, replayed so timers fire at the same recorded instants, wall-clock-independent; armed deadlines survive checkpoint compaction |
| **Invoked services** | `WithServiceRegistry` + `Handle.RunService` | `JournalServiceResult`; the service runs once, then recovery replays its result through the kernel's settle seam |
| **Child-machine actors** | `WithActorPalette` + `Handle.DeliverToActor` | `JournalActorMessage`; the behavior runs once, then recovery re-fires the recorded parent transition |
| **Domain effects** | `WithEffectHandler` | dispatch set, applied exactly once via deterministic `EffectID` dedup |
| **Invoked services** | `WithServiceRegistry` + `Handle.RunService` | `JournalServiceResult`; a recorded result replays through the kernel's settle seam without re-running the service, and an unrecorded one (a context canceled during the service, a failed `Append`, a crash) runs again when the host calls `RunService` |
| **Child-machine actors** | `WithActorPalette` + `Handle.DeliverToActor` | `JournalActorMessage`; recovery re-fires a recorded parent transition; a settled actor doesn't re-run, and a running one restarts from its initial state on any `Recover` |
| **Domain effects** | `WithEffectHandler` | dispatch set, a step's effects applied exactly once via deterministic `EffectID` dedup, with the exceptions under Exactly-once effects |

Use `WithCheckpointEvery(n)` to tune how often a full snapshot is written (a
shorter interval bounds recovery replay; a longer one cuts checkpoint cost).
Expand Down Expand Up @@ -136,14 +138,27 @@ opts in by implementing `HistoryStore` (the in-tree `MemStore` does so under
`WithHistory`); `StateAt` falls back to `Load` (latest checkpoint plus tail)
otherwise.

## Handles across goroutines

`Fire`, `Tick`, `RunService` and `DeliverToActor` are safe to call from different
goroutines: each holds the Handle's lock for the whole step. A host callback must not
call back into its Handle. `Instance()` is outside the lock.

A step that may not have been recorded, a failed effect dispatch, or a `Tick`, `RunService`
or `DeliverToActor` whose context ended partway, fails the Handle with `ErrHandleFailed`.
`Recover` a new Handle with the same options to continue, unless the cause is `ErrFenced`:
then a newer owner holds the instance, so stop. `Recover` retries an unapplied effect, and
returns `ErrEffectDispatch` while the handler still fails.

## Caveat: serialized payloads

Events, service done-data, actor done-data, and actor messages are recorded as their
JSON form. A parent reducer that type-asserts a non-JSON Go type from `AssignCtx.Event`
observes the JSON-decoded shape on a replayed `onDone`. A `WithEventData` payload is
JSON-decoded live as well as on replay, so a reducer gets the same value both ways: a
struct is a map and an integer a `float64`. An error stays an error with the same
message, with any invalid UTF-8 replaced. `Handle.Fire` refuses a payload it can't
observes the JSON-decoded shape of actor done-data on a replayed `onDone`. A
`WithEventData` payload and a service's result are JSON-decoded live as well as on replay,
so a reducer gets the same value both ways: a struct is a map and an integer a `float64`,
which loses precision above 2^53. An error stays an error with the same message, with any
invalid UTF-8 replaced, and `errors.Is` no longer matches it. `Handle.Fire` refuses a payload it can't
record, or one over `WithMaxEventDataBytes` (8 MiB by default), with
`ErrEventDataEncoding`, and nothing applies. A
typed-codec option to carry the concrete Go value across the journal boundary is
Expand Down
Loading
Loading