Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ let them bring their own? If it forces, redesign it as a seam with a no-op defau

Requirements:

- **Go**: 1.27.3, the version every `go.mod` and `go.work` declares. CI reads it from
- **Go**: 1.27.4, the version every `go.mod` and `go.work` declares. CI reads it from
`go.work`.
- **[Mage](https://magefile.org)**: the build tool. Install with
`go install github.com/magefile/mage@latest`, or run targets via
Expand Down
2 changes: 1 addition & 1 deletion cluster/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ and this module adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.

### Changed

- Requires Go 1.27.3.
- Requires Go 1.27.4.

### Added

Expand Down
2 changes: 1 addition & 1 deletion cluster/go.mod
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
module github.com/stablekernel/crucible/cluster

go 1.27.3
go 1.27.4

replace github.com/stablekernel/crucible/state => ../state

Expand Down
2 changes: 1 addition & 1 deletion cmd/crucible/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ versioned independently of the `state` module.

### Changed

- Requires Go 1.27.3.
- Requires Go 1.27.4.

### Added

Expand Down
2 changes: 1 addition & 1 deletion cmd/crucible/go.mod
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
module github.com/stablekernel/crucible/cmd/crucible

go 1.27.3
go 1.27.4

require (
github.com/stablekernel/crucible/gen v0.1.0
Expand Down
39 changes: 33 additions & 6 deletions docs/src/content/docs/durable/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,15 +125,42 @@ Time-travel needs the full record history through the target step. A `Store` opt
in by implementing `HistoryStore` (the in-tree `MemStore` does so under
`WithHistory`); `StateAt` otherwise falls back to the latest checkpoint plus tail.

## Handles across goroutines

`Fire`, `Tick`, `RunService` and `DeliverToActor` are safe to call from different
goroutines: each holds the Handle's lock while it applies to the kernel and records the
step. Effects dispatch after the lock is released, so an effect handler can call back
into its Handle. A service, guard, action, reducer or actor behavior runs under the lock,
so it can't act on the Handle itself: to act on the Handle when a step happens, an action
returns an effect, whose handler runs after the lock. If one calls back during a live
step with the step's context, as a service gets and a guard, action or reducer finds in
its input's `Ctx`, the call returns `ErrReentrantStep` instead of waiting. A call back
without that context deadlocks, and so does one under middleware that passed down a
context not derived from the step's. A step waiting for the lock stops when its context
ends. Read the instance with `View`, which
takes the lock; `Instance()` is outside it.

A step that may not have been recorded fails the Handle with `ErrHandleFailed`. So does an
effect handler that returns an error or panics. Before `Recover`, call `Handle.Drain`,
which waits until no step of the failed Handle is still recording and no effect is still
being sent, so the recovered Handle loads every recorded step and can't send an effect
twice. Give `Drain` a context of its own, not the failed call's, and don't `Recover`
until it returns nil: an error means it didn't finish draining. A step that was recording when another goroutine failed the Handle returns
`ErrHandleFailed` saying it was recorded; `Recover` sends any of its effects. To move a healthy Handle's instance to a new Handle, stop
calling the old one and let its calls return first: a step can start after `Drain`
returns. `Recover` a new Handle with the same options to continue, unless the cause is `ErrFenced`: then a newer owner holds the
instance, so stop.

## A note on serialized payloads

Events, service done-data, actor done-data, and actor messages are recorded as their
JSON form. A parent reducer that type-asserts a non-JSON Go type from `AssignCtx.Event`
observes the JSON-decoded shape on a replayed `onDone`. A `WithEventData` payload is
JSON-decoded live as well as on replay, so a reducer gets the same value both ways: a
struct is a map and an integer a `float64`. An error stays an error with the same
message, with any invalid UTF-8 replaced. `Handle.Fire` refuses a payload it can't
decoded from its JSON form live as well as on replay, so a reducer gets the same value
both ways. `WithEventDataDecoder` decodes it as the type the caller passed, chosen per
event, for example `durable.DecodeAs[Order]`. Without one, a payload decodes as JSON's own
form, so `Handle.Fire` refuses a struct or an integer rather than hand the reducer a map
or a `float64`. An error stays an error with the same message, with any invalid UTF-8
replaced. `Handle.Fire` refuses a payload that decodes as a different type, one it can't
record, or one over `WithMaxEventDataBytes` (8 MiB by default), with
`ErrEventDataEncoding`, and nothing applies. A
typed-codec option to carry the concrete Go value across the journal boundary is
reserved for a later, additive change.
`ErrEventDataEncoding`, and nothing applies.
69 changes: 63 additions & 6 deletions durable/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,14 +25,62 @@ and this module adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.
a refused payload, and `ErrRecordTooLarge` a Record too large for a `FileStore` line.
`Record.EventData` and `Record.EventDataError` hold the payload.

- **`ErrHandleFailed`** reports a Handle out of step with its journal.

- **`Handle.View`** reads the instance under the Handle's step lock.

- **`Handle.Drain`** waits until no step of the Handle is still recording and no effect
is still being sent. Call it before `Recover` after `ErrHandleFailed`, with a context
of its own, and don't `Recover` until it returns nil. On a healthy Handle it covers
only work already under way.

- **`ErrReentrantStep`** reports a call back into a Handle that carries the context one
of its steps passed to a callback: a service's context, or the `Ctx` a guard, action
or reducer gets (#245). A call back without that context deadlocks instead.

- **`StepReporter`**, implemented by `MemStore` and `FileStore`, lets `Recover` number the
next step past every recorded one after a checkpoint from before checkpoints carried
it. Without it, such a checkpoint followed by tick barriers made every `Append`
conflict.

- **`WithEventDataDecoder`** decodes a `WithEventData` payload as the caller's type,
given the event it rode with. `DecodeAs[T]` is the usual body of each case.

### Changed

- Requires Go 1.27.3.
- **A `WithEventData` payload reaches `Assign` decoded, live as well as on replay.** A
struct arrives as a map and an integer as a `float64`, which loses precision above
2^53, so an assertion to the original
type now fails at `Fire`, not at `Recover`. An error arrives as an error with the same
message, any invalid UTF-8 replaced, and `errors.Is` no longer matches it.
- Requires Go 1.27.4.
- **A Handle's steps are serialized across goroutines.** `Fire`, `Tick`, `RunService` and
`DeliverToActor` hold a lock while they apply to the kernel and record the step, so a
host's timer loop no longer loses acknowledged steps. Effects dispatch after the lock
is released, so an effect handler can still call back into its Handle; a checkpoint
waits for every earlier step's effects to dispatch. A service, guard, action, reducer
or actor behavior runs under the lock, so it can't act on the Handle itself; an action
reaches the Handle by returning an effect. A service, guard, action or reducer that
calls back with the context its step passed gets the new `ErrReentrantStep`. A step
waiting for the lock returns its context's error once the context ends, and changes
nothing. The new `Handle.View` reads the instance under the lock; `Instance()` is
outside it.
- **A failed effect dispatch fails the Handle**, and so does an effect handler that
panics, which now returns `ErrEffectDispatch` instead of panicking out of `Fire`. The
effect stays unmarked in the journal, so no later checkpoint compacts it away, and
`Recover` dispatches it again.
- **A step that may not have been recorded fails the Handle.** That call and every later
one return `ErrHandleFailed`, wrapping the cause. A step that was recorded while
another goroutine failed the Handle also returns `ErrHandleFailed`, saying the step was
recorded; `Recover` sends any of its effects. Call `Drain` before `Recover`, so a
step another goroutine is still recording is loaded, and an effect it is still sending
isn't sent twice. `Recover` a new Handle
with the same options to continue. A service whose result wasn't recorded is in flight again, so
`RunService` runs it a second time. An `InvokeActor` actor restarts from its initial
state, and a `Spawn` actor doesn't return. If the cause is `ErrFenced`, a newer owner
holds the instance: stop.
- **A `WithEventData` payload reaches `Assign` decoded, live as well as on replay.**
`WithEventDataDecoder` decodes it as the caller's type, chosen per event, with
`DecodeAs[T]`. A payload that decodes as a different type than the caller passed
refuses the `Fire` with `ErrEventDataEncoding`, so without a decoder a struct or an
integer is refused rather than reaching `Assign` as a map or a `float64`. An error
arrives as an error with the same message, any invalid UTF-8 replaced, and `errors.Is`
no longer matches it.
- **`Handle.Fire` can refuse a payload.** One with no JSON form that decodes again, or
over `WithMaxEventDataBytes` (8 MiB by default), returns the new `ErrEventDataEncoding`
before the kernel runs. Fires of that size used to pass, because the payload wasn't
Expand All @@ -59,6 +107,15 @@ and this module adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.

### Fixed

- **A failed checkpoint no longer makes the next step conflict.** `Tick`, `RunService`
and `DeliverToActor` now advance the next step as soon as the append lands.
- **A deferred checkpoint survives an unrelated caller's cancel.** A checkpoint that
waited for an earlier step's effects to dispatch, and was canceled by the context of
the caller whose step finished last, isn't reported to that caller, since its own
step succeeded, and the next step writes it; a failure of that retry is returned to
the step that tried it. A retry the Store refuses as not advancing means the first
write landed, and counts as written. Any other checkpoint failure is returned, and
after that only the next step the policy makes due tries again.
- **The next step survives a checkpoint that compacts tick barriers.** A checkpoint
now records the step that follows it, in a `nextStep` field of its envelope. Recovery
used the Trace count, which runs low past a tick barrier, so the next Fire could reuse a
Expand Down
39 changes: 33 additions & 6 deletions durable/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,18 +136,45 @@ opts in by implementing `HistoryStore` (the in-tree `MemStore` does so under
`WithHistory`); `StateAt` falls back to `Load` (latest checkpoint plus tail)
otherwise.

## Handles across goroutines

`Fire`, `Tick`, `RunService` and `DeliverToActor` are safe to call from different
goroutines: each holds the Handle's lock while it applies to the kernel and records the
step. Effects dispatch after the lock is released, so an effect handler can call back
into its Handle. A service, guard, action, reducer or actor behavior runs under the lock,
so it can't act on the Handle itself: to act on the Handle when a step happens, an action
returns an effect, whose handler runs after the lock. If one calls back during a live
step with the step's context, as a service gets and a guard, action or reducer finds in
its input's `Ctx`, the call returns `ErrReentrantStep` instead of waiting. A call back
without that context deadlocks, and so does one under middleware that passed down a
context not derived from the step's. A step waiting for the lock stops when its context
ends. Read the instance with `View`, which
takes the lock; `Instance()` is outside it.

A step that may not have been recorded fails the Handle with `ErrHandleFailed`. So does an
effect handler that returns an error or panics. Before `Recover`, call `Handle.Drain`,
which waits until no step of the failed Handle is still recording and no effect is still
being sent, so the recovered Handle loads every recorded step and can't send an effect
twice. Give `Drain` a context of its own, not the failed call's, and don't `Recover`
until it returns nil: an error means it didn't finish draining. A step that was recording when another goroutine failed the Handle returns
`ErrHandleFailed` saying it was recorded; `Recover` sends any of its effects. To move a healthy Handle's instance to a new Handle, stop
calling the old one and let its calls return first: a step can start after `Drain`
returns. `Recover` a new Handle with the same options to continue, unless the cause is `ErrFenced`: then a newer owner holds the
instance, so stop.

## Caveat: serialized payloads

Events, service done-data, actor done-data, and actor messages are recorded as their
JSON form. A parent reducer that type-asserts a non-JSON Go type from `AssignCtx.Event`
observes the JSON-decoded shape on a replayed `onDone`. A `WithEventData` payload is
JSON-decoded live as well as on replay, so a reducer gets the same value both ways: a
struct is a map and an integer a `float64`. An error stays an error with the same
message, with any invalid UTF-8 replaced. `Handle.Fire` refuses a payload it can't
decoded from its JSON form live as well as on replay, so a reducer gets the same value
both ways. `WithEventDataDecoder` decodes it as the type the caller passed, chosen per
event, for example `durable.DecodeAs[Order]`. Without one, a payload decodes as JSON's own
form, so `Handle.Fire` refuses a struct or an integer rather than hand the reducer a map
or a `float64`. An error stays an error with the same message, with any invalid UTF-8
replaced. `Handle.Fire` refuses a payload that decodes as a different type, one it can't
record, or one over `WithMaxEventDataBytes` (8 MiB by default), with
`ErrEventDataEncoding`, and nothing applies. A
typed-codec option to carry the concrete Go value across the journal boundary is
reserved for a later, additive change.
`ErrEventDataEncoding`, and nothing applies.

## Performance

Expand Down
27 changes: 20 additions & 7 deletions durable/actor.go
Original file line number Diff line number Diff line change
Expand Up @@ -98,25 +98,39 @@ type actorMessagePayload struct {
// JournalActorMessage correlated by the actor id so recovery replays it without
// re-instantiating the actor.
//
// It returns whether the actor was found running (false for a ref that names no
// live actor — already settled, stopped, or never spawned), and a persistence
// failure as the error. The kernel transitions the delivery drove do not
// themselves fail.
// It returns false for a ref that names no live actor (already settled, stopped, or
// never spawned), and false with the error when the Handle had already failed or ctx
// has ended. If the actor received the event but the delivery couldn't be recorded or
// appended, it returns true and ErrHandleFailed: the Handle has failed. If another
// goroutine failed the Handle while this step was recording, it returns true and
// ErrHandleFailed saying the step was recorded. A failed checkpoint returns true and
// that error, and the Handle carries on. The kernel
// transitions the delivery drove do not themselves fail. When a delivery wasn't
// recorded, on a recovered Handle an InvokeActor actor restarts from its initial state,
// and a Spawn actor doesn't return. Resending a recorded delivery repeats
// any message it sends the parent.
func (h *Handle[S, E, C]) DeliverToActor(ctx context.Context, ref state.ActorRef, event any) (bool, error) {
if err := h.acquire(ctx); err != nil {
return false, err
}
defer h.release()
if err := h.refuse(ctx); err != nil {
return false, err
}
if h.actors == nil {
return false, fmt.Errorf("durable: no actor palette wired for %q (supply WithActorPalette)", h.id)
}

before := len(h.inst.History())
delivered := h.actors.Deliver(ctx, ref, event)
delivered := h.actors.Deliver(h.stepContext(ctx), ref, event)
if !delivered {
return false, nil
}

output, settled, settleErr := h.actorOutcome()
entries, err := recordActorTransitions(ref.ID, h.inst.History(), before, output, settled, settleErr)
if err != nil {
return true, fmt.Errorf("durable: recording actor %q result for %q: %w", ref.ID, h.id, err)
return true, h.fail(fmt.Errorf("durable: recording actor %q result for %q: %w", ref.ID, h.id, err))
}
if h.actorBuf != nil {
*h.actorBuf = append(*h.actorBuf, entries...)
Expand All @@ -127,7 +141,6 @@ func (h *Handle[S, E, C]) DeliverToActor(ctx context.Context, ref state.ActorRef
if err := h.persistStep(ctx, step, &rec); err != nil {
return true, err
}
h.nextStep++
return true, nil
}

Expand Down
6 changes: 3 additions & 3 deletions durable/doc.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,9 @@
// Payloads are serialized at the journal boundary. Event, service done-data, actor
// done-data, and actor messages are recorded as their JSON form; a parent reducer that
// type-asserts a non-JSON Go type from AssignCtx.Event observes the JSON-decoded shape
// on a replayed onDone. A WithEventData payload is JSON-decoded live as well as on
// replay, so a reducer gets the same value both ways. A typed-codec option to carry the
// concrete Go value across the boundary is reserved for a later, additive change.
// on a replayed onDone. A WithEventData payload is decoded from its JSON form live as
// well as on replay, so a reducer gets the same value both ways; WithEventDataDecoder
// decodes it as the type the caller passed.
//
// The kernel stays pure and dependency-free. The durable module is host-side:
// database drivers, cloud SDKs, and I/O libraries live in the caller or in out-
Expand Down
11 changes: 6 additions & 5 deletions durable/effect.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,9 @@ import (
// once) and, because it is then marked, never again.
// - Crash AFTER dispatch (id marked): recovery re-emits the effect but finds its
// id already marked and skips it — so it is not redispatched (exactly once).
// - Handler error: the effect is left un-marked and the error surfaces wrapped in
// ErrEffectDispatch, so a later recovery retries it (at-least-once until it
// succeeds; exactly-once thereafter).
// - Handler error: the effect is left un-marked, and the Handle fails with
// ErrHandleFailed wrapping ErrEffectDispatch, so Recover retries it
// (at-least-once until it succeeds; exactly-once thereafter).
//
// The dedup set is held by the Store (DispatchStore) so it shares the Record's
// durability and survives checkpoint compaction — a delayed redispatch of an
Expand All @@ -70,7 +70,8 @@ import (
// EffectHandler applies one emitted domain effect, identified by its stamped
// EffectID. The Runner calls it exactly once per EffectID over the lifetime of an
// instance (see WithEffectHandler). A non-nil error leaves the effect un-marked
// for a later retry and is surfaced to the caller wrapped in ErrEffectDispatch.
// for Recover to retry, and fails the Handle with ErrHandleFailed wrapping
// ErrEffectDispatch and the error.
type EffectHandler func(ctx context.Context, effectID string, effect state.Effect) error

// DispatchStore is the dedup seam for idempotent effect dispatch: the Store
Expand Down Expand Up @@ -226,7 +227,7 @@ func (r *Runner[S, E, C]) dispatch(ctx context.Context, id InstanceID, des []dis
continue
}
if err := r.cfg.effectHandler(ctx, de.id, de.effect); err != nil {
return fmt.Errorf("%w: effect %q for %q: %v", ErrEffectDispatch, de.id, id, err)
return fmt.Errorf("%w: effect %q for %q: %w", ErrEffectDispatch, de.id, id, err)
}
if err := ds.MarkDispatched(ctx, id, de.id); err != nil {
return fmt.Errorf("durable: marking effect %q dispatched for %q: %w", de.id, id, err)
Expand Down
Loading
Loading