Skip to content

fix(durable): lock the Handle and refuse steps after a failed Append - #243

Merged
reka18 merged 10 commits into
mainfrom
fix/durable-handle-lock-failed-append
Nov 11, 2026
Merged

reka18 merged 10 commits into
mainfrom
fix/durable-handle-lock-failed-append

Conversation

@reka18

@reka18 reka18 commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

What this change does

Two #214 fixes in the same code path, rebased onto main now that #232 and #245 have merged. This replaces #233, which GitHub couldn't reopen after a rebase. It also folds in the review of #233; my comment there lists each finding and where it landed.

  • Handle lock. Fire, Tick, RunService and DeliverToActor hold a lock while they apply to the kernel and record the step, so they're safe from different goroutines. A step waiting for the lock returns its context's error once the context ends, and changes nothing. This is the contract in my answer to question 10 on #214, which is a working answer, not a ruling.
  • Effects dispatch outside the lock. An effect handler can call back into its Handle. A checkpoint waits until every earlier step's effects have dispatched, so it never compacts one still being sent. A failed dispatch, or a handler that panics, fails the Handle with ErrEffectDispatch and leaves the effect for Recover to send again.
  • Failed Append. A step that applied to the kernel but may not have been recorded fails the Handle. That call and every later one return the new ErrHandleFailed, wrapping the cause. Call the new Handle.Drain, with a context of its own, then Recover a new Handle once it returns nil. Drain waits for a step still writing its Record, then for dispatch. A step recorded while another goroutine failed the Handle returns ErrHandleFailed saying it was recorded. nextStep now advances right after the append, so a failed checkpoint no longer makes the next step conflict.
  • Calling back in. A service, guard, action or reducer runs under the lock. One that calls back into its Handle with the context its step passed (a service's context, or the Ctx from feat(state): pass the step's context to callbacks; skip CI on drafts #245) gets the new ErrReentrantStep at once. A call without that context still deadlocks; the docs say so, and say to return an effect instead.
  • One refusal check. Each step's entry check, refuse(ctx), returns ErrHandleFailed for a failed Handle and ctx.Err() for an ended context.
  • View reads the instance under the lock. Instance() still returns it outside.
  • Recovery after an old checkpoint. MemStore and FileStore implement the new StepReporter, so Recover numbers the next step past every recorded one. A checkpoint from before fix(durable): carry the next step through a checkpoint #232 followed by a tick barrier no longer makes every Append conflict.
  • Event data reaches Assign decoded, live as on replay. WithEventDataDecoder decodes a payload as the caller's type, chosen per event, with DecodeAs[T]. A payload that decodes as a different type than the caller passed refuses the Fire with ErrEventDataEncoding, rather than reaching Assign as a map.
  • Deferred checkpoints. A checkpoint that waited for an earlier step's dispatch, and was canceled by the context of the caller whose step finished last, isn't reported to that caller, since its own step succeeded; the next step writes it. Any other checkpoint failure is returned, and the next step the policy makes due tries again.
  • FileStore.Append clones the record once.
  • Go 1.27.4. Every go.mod and go.work moves to Go 1.27.4, which fixes GO-2026-6859, an HTTP/1 server crash in net/http that govulncheck now fails every PR on. No dependency changes.

Please check these:

  • Assumption: the per-step Append stays inline, on the session's single owner goroutine. If it moves off that goroutine, the lock scope and this failure handling change.
  • A running service holds the lock, so a slow service blocks other steps. state.ServiceRunner runs and settles in one call, so I couldn't narrow it here.
  • If the cause is ErrFenced, a newer owner holds the instance, so Recover won't help. The docs say to stop.
  • After a failed Append, a service whose result wasn't recorded is in flight again on the recovered Handle, so RunService runs it a second time. An InvokeActor actor restarts from its initial state, and a Spawn actor doesn't return.

Known and left open:

  • A Store that panics mid-step releases the lock but skips the failure latch, if the host recovers the panic.
  • Stateless Runner.Fire calls on one id aren't serialized. The doc says so.
  • A refused Fire leaves a Trace only in the live instance, so a later snapshot can differ byte for byte.
  • Start doesn't pass its context to the callbacks that settle the initial state, and cluster.WithActorBehaviors has no form that takes a context. Neither can reach a Handle's lock today.

Why

A host's timer loop racing Fire lost acknowledged steps, and a failed Append left the live instance ahead of the journal.

Relates to: #214, #245

Evidence

On GOTOOLCHAIN=go1.27.4:

Check Result
go test -race ./state/... ./durable/... ./cluster/... Pass
go test ./... in examples/dispatch and e2e Pass
Pinned golangci-lint on durable 0 issues
govulncheck in each module CI scans No vulnerabilities
go test ./... in gen and transport Pass

The new tests in handle_lock_test.go, handle_failure_test.go, runner_test.go and event_data_test.go cover the timer loop racing Fire, a failed Append on each step, an ended context, a deadline in the lock queue, a panic under the lock, dispatch outside the lock and a checkpoint waiting for it, a failed or panicking effect handler, Drain, View, a service, guard, action or reducer calling back in, StepReporter recovery and event data decoding. Each fails with the code it covers removed. The timer-loop test is reliable under -race; a plain run can pass by luck.

Checklist

  • Commits are signed off (git commit -s) per the DCO
  • Conventional commit messages (type: subject)
  • mage check passes locally (lint + race tests + govulncheck): not run as a whole; the parts above were run individually
  • Tests added/updated for the change
  • Public API changes documented (godoc) and, if applicable, noted for the changelog

@reka18 reka18 added ready All ACR dimensions confirmed - clear to merge confirmed:correctness ACR clean: correctness confirmed:regression-safety ACR clean: regression-safety confirmed:concurrency-discipline ACR clean: concurrency-discipline confirmed:replay-determinism ACR clean: replay-determinism confirmed:api-compatibility ACR clean: api-compatibility confirmed:missed-context ACR clean: missed-context confirmed:human-readability ACR clean: human-readability labels Nov 10, 2026
@reka18
reka18 requested a review from joshua-temple November 10, 2026 04:32
@reka18
reka18 added this pull request to stack #244 November 10, 2026 14:47
@reka18

reka18 commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

Findings from Temple's review of #233

Temple reviewed #233, the PR this one replaces. These are his findings and where each one is handled. Line numbers are from #233's old head (67d9182).

High

  • Effect handlers that call back into the Handle deadlock (runner.go:457). This is real, and this PR causes it. Fix in this PR: the lock will cover only the kernel step and the Append, and effects will dispatch after it's released. A checkpoint won't compact past an effect that hasn't been dispatched. A call back in on the step's own context, from a service function or actor behavior that has to stay under the lock, will return an error instead of hanging.
  • A checkpoint saved before fix(durable): carry the next step through a checkpoint #232 can recover with the wrong step number (runner.go:783). This is real. fix(durable): carry the next step through a checkpoint #232 added the NextStep field. A checkpoint from before fix(durable): carry the next step through a checkpoint #232 doesn't carry it, so recovery falls back to counting Traces. A tick barrier or an actor delivery throws that count off, the next Append conflicts, and every later Recover hits the same conflict. Fix in this PR: the built-in Stores already track the highest recorded step, so recovery will ask them for it. It will fall back to the Trace count only for a Store that can't report it.
  • A Tick whose context ends partway replays differently from the live run (runner.go:487). This PR lists it as known. Fixed in fix(durable): don't journal a step whose context ended partway #234: a Tick, RunService or DeliverToActor whose context ends partway fails the Handle and records nothing, so the recovered instance matches the journal.

Medium

  • A step waiting for the lock ignores its context deadline (:104). Fix in this PR: the lock becomes one a waiting step can leave, so it returns ctx.Err() once its context ends.
  • Live Fire passes event data as decoded JSON, not the caller's Go value (:853). This came from fix(durable): journal WithEventData payloads for recovery #230. It was done so a live step and its replay see the same value, but a reducer that expects its own struct type gets a map. To fix here: let the Runner decode event data into the caller's type, so live and replay both see that struct. I'll post the exact shape before it lands.
  • Instance() isn't covered by the lock (:270). The doc warns about it, but it's an easy race. Fix in this PR: a locked read on the Handle, so a host can read the instance without racing a step.

Low

A fresh review round runs on this PR before it changes, covering correctness, concurrency and replay determinism. The PRs above it will be rebased once it merges.

@reka18

reka18 commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

Review of the rebased change (494c78a..2f8b57c)

The change is rebased onto main after #232 merged, plus one fix commit (2f8b57c). All seven dimensions are confirmed, so the PR is ready.

The reviewers (AI agents, one per dimension, each starting with a clean context) could only read code: they traced the code paths but couldn't run tests. I ran the tests and the mutation checks myself.

Dimension Round 1 Round 2 Status
Correctness Blockers, all pre-existing (see below) 0 blockers Confirmed
Concurrency discipline Blockers, all pre-existing 0 blockers Confirmed
Replay determinism Blockers, all pre-existing 1 blocker, pre-existing, fixed in #236 Confirmed (my ruling)
Regression safety 1 blocker 0 blockers Confirmed
API compatibility 0 blockers Not rerun Confirmed
Missed context 0 blockers Not rerun Confirmed
Human readability 0 blockers Not rerun Confirmed

Fixed in 2f8b57c

  • Blocker: the RunService and DeliverToActor docs said a step that wasn't recorded returns false. It returns true along with ErrHandleFailed, and a failed checkpoint returns true along with that error. The docs now say so.
  • Tests: RunService and DeliverToActor had no tests for the failure latch or the ended-context guard. Four tests in handle_failure_test.go now cover them, and each fails when its guard is removed. The Tick ended-context test now also checks that nothing was appended.
  • Docs:
    • The Handle doc says a context that ends during the Store write can also fail the Handle.
    • A refused Fire now "journals nothing".
    • A comment at persistStep states the assumption that Append runs inline.
    • The "kernel transition itself does not fail" claim on RunService is removed.
    • The sentinel group comment covers the Handle.

Rejected: pre-existing on main, fixed higher in the stack

These come from code this PR doesn't change. Fixing them here would mean fixing more than the bottom PR of the stack.

Pre-existing, possibly not tracked yet

  • When Fire spawns an actor and the spawn fails, the parent's error transition runs on the live instance but isn't journaled.
  • When a delivery to an actor drives a parent transition into a state with an after timer or an invoked service, that timer or service isn't started.
  • A clock read from outside a step can be counted in the step that's open at the time.
  • DeliverToActor writes an empty Record when the delivery drove no parent transition.

Non-blockers left as they are

  • ActorRef reads outside the lock, and its docs don't say so the way the Instance() docs do.
  • The failed-Append path of Tick itself has no direct test. The tests cover the same path through Fire, RunService and DeliverToActor.
  • Tick could call persistStep instead of repeating its steps inline.
  • The timer-loop race test triggers the race by chance, though it reliably catches the race when it happens.
  • Recover's doc doesn't repeat the rule against overlapping a live Handle.
  • A Tick that never stops re-arming now holds the lock indefinitely.

Verification

On GOTOOLCHAIN=go1.27.3:

  • go test -race ./... passes in durable.
  • go test ./... passes in state and examples/dispatch.

Comment thread durable/runner.go Resolved
@reka18

reka18 commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

#245 passes the step's context to guards, actions, reducers and actor factories. Today only a service's call back into its own Handle is caught with ErrReentrantStep, because only a service gets the step's context; the same call from a guard, action, reducer or actor factory deadlocks. With #245, one that passes in.Ctx back gets ErrReentrantStep too.

@reka18
reka18 marked this pull request as draft November 11, 2026 14:57
Fire, Tick, RunService and DeliverToActor now hold a lock for the whole step,
so a host's timer loop no longer loses acknowledged steps. A step that applied
to the kernel but wasn't appended leaves the instance ahead of the journal, so
the Handle refuses further steps with ErrHandleFailed until it's recovered.

Relates to #214.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…rToActor

RunService and DeliverToActor refuse steps after a failed Append and
return an ended context's error, but only Fire and Tick were tested.
New tests fail an Append under each and end the context before each,
and check nothing ran or was journaled. The Tick context test now also
checks that nothing was appended.

The RunService and DeliverToActor docs said a failed step returns
false; they return true with ErrHandleFailed once the service settled
or the actor received the event. The docs now say so, name a context
that ends during the Store write as a possible cause, and note that the
latch relies on the Append running inline.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…w gaps

An effect handler that called back into its own Handle deadlocked, because
Fire held the step lock through dispatch. The lock now covers only the
kernel step and the Record append. Effects dispatch after it's released,
and a checkpoint waits until every step at or before it has dispatched, so
compaction never drops an effect still being sent. A failed dispatch fails
the Handle, so Recover sends the effect again.

Code that has to run under the lock gets a marked context: a service that
calls back in with it gets ErrReentrantStep instead of deadlocking. A step
waiting for the lock stops when its context ends. Handle.View reads the
instance under the lock.

Also:
- WithEventDataDecoder and DecodeAs decode a WithEventData payload as the
  caller's type, live and on replay. A payload that decodes as another
  type refuses the Fire.
- StepReporter lets Recover number past every recorded step after a
  checkpoint from before checkpoints carried the next step. MemStore and
  FileStore implement it.
- FileStore.Append encodes a Record once.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…ailing

Review of the outbox change found gaps:
- Fire released the step lock without defer, so a panic in host code
  under the lock left it held. It now releases however the step ends.
- An effect handler that panicked left its step dispatching forever,
  which silently stopped every later checkpoint. A panic now fails the
  Handle with ErrEffectDispatch, like a returned error.
- A call that fails the Handle returns only once no effect is still
  being sent, so a host that recovers right away can't send one twice.
  A step that held the lock while another goroutine failed the Handle
  leaves its effects for Recover instead of sending them.
- refuse takes the context, so every step runs one check after taking
  the lock: the Handle hasn't failed, and the context hasn't ended.
- A nil payload stays nil under a typed decoder, and ErrEffectDispatch
  wraps the handler's error.
- Docs: the wait after a failure, handler failures failing the Handle,
  dispatch order for a step fired from a handler, StepReporter errors.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
A call that failed the Handle waited until no effect was still being
sent. An effect handler that called back with a context other than its
own, and failed the Handle, waited on its own dispatch forever, and so
did every later caller. Calls that fail the Handle now return at once.
Handle.Drain waits until no effect is still being sent; call it before
Recover. Called from inside a step or an effect handler, it returns
ErrReentrantStep instead of waiting.

Docs: an action reaches the Handle by returning an effect; a guard,
action, reducer or actor behavior that calls back without the step's
context deadlocks. Fire's doc covers a failed checkpoint. The payload
cap measures an error's JSON form, and a Store panic during dispatch is
reported as a dispatch panic, not a handler's.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
Once a Handle has failed, no step can start sending effects, so Drain covers every one. On a healthy Handle a step still recording can start sending after Drain returns, so the docs say to stop calling it before moving its instance to a new Handle. Also: a typed-nil payload is refused as a type mismatch.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
@reka18
reka18 force-pushed the fix/durable-handle-lock-failed-append branch 2 times, most recently from 560078a to 2c2eb92 Compare November 11, 2026 15:03
…s Ctx

With #245, a guard, action or reducer gets the step's context as its
input's Ctx, so a call back into the Handle with it returns
ErrReentrantStep instead of deadlocking. Say so in the docs, including
that a call back without the step's context, or under middleware that
passed down an unrelated one, still deadlocks. Cover Fire, an eventless
transition, middleware, Tick, RunService and DeliverToActor in a test.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
@reka18
reka18 force-pushed the fix/durable-handle-lock-failed-append branch from 2c2eb92 to 4626b47 Compare November 11, 2026 15:23
@reka18
reka18 marked this pull request as ready for review November 11, 2026 15:23
@reka18
reka18 requested a review from joshua-temple November 11, 2026 15:23
govulncheck now fails every PR on GO-2026-6859, an HTTP/1 server crash
in net/http that Go 1.27.4 fixes. It's reached through grpc's server in
transport. Every go.mod and go.work now declares Go 1.27.4; CI reads the
version from go.work. No dependency changes.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
@reka18 reka18 added wait:replay-determinism ACR pending: replay-determinism wait:api-compatibility ACR pending: api-compatibility wait:missed-context ACR pending: missed-context wait:human-readability ACR pending: human-readability and removed ready All ACR dimensions confirmed - clear to merge confirmed:regression-safety ACR clean: regression-safety confirmed:replay-determinism ACR clean: replay-determinism confirmed:api-compatibility ACR clean: api-compatibility confirmed:missed-context ACR clean: missed-context confirmed:human-readability ACR clean: human-readability labels Nov 11, 2026
@reka18

reka18 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor Author

Temple's Drain and checkpoint findings

Both are fixed in b9e6074.

Medium: Drain could return before a step's write finished. Drain now waits for the step lock before it waits for dispatch, so a step that was mid-Append when another goroutine failed the Handle finishes first, and Recover loads it. That step's caller used to get a bare ErrHandleFailed, or success if the step had no effects. Now it gets ErrHandleFailed saying the step was recorded and that Recover sends any of its effects, on Fire, Tick, RunService and DeliverToActor. The docs now say to give Drain a context of its own and not to Recover until it returns nil. TestHandle_Drain_WaitsForAStepStillRecording and TestHandle_StepRecordedAsTheHandleFails_SaysSo cover it, the second for all four step methods.

Low: a deferred checkpoint used an unrelated caller's context. When the caller whose step finished last cancels, it's no longer told about a checkpoint it didn't make due, since its own step succeeded, and the next step writes that checkpoint. Before, the checkpoint was dropped and the error went to that caller. A retry the Store refuses as not advancing counts as written, because the first write landed. Any other checkpoint failure is still returned, and only the next step the policy makes due tries again, so other steps aren't told about a checkpoint they didn't make due. Five tests in handle_lock_test.go cover these cases.

Each fix is mutation-checked: with it removed, its test fails.

How the review changed

Our earlier rounds missed both bugs, so this round added two guards. The concurrency reviewer had to build a table of what else can be in flight against each changed method and whose context any deferred work uses. A new probe reviewer wrote and ran tests in a throwaway worktree to break the change. The probe found three more problems the read-only reviewers hadn't, each with a failing test, and all are fixed here:

  • The "recorded" error reached only Fire with effects.
  • A checkpoint nobody could cancel blocked Drain.
  • A checkpoint the Store wrote but reported as failed made every later step return an error.
Dimension Round 1 Round 2 Round 3 Status
Concurrency discipline 2 blockers 0 blockers Confirmed
Correctness 1 blocker 0 blockers Confirmed
Probe (writes and runs tests) 1 blocker 1 blocker 0 blockers Confirmed

Two non-blockers from the last probe round were Ray's call, and both are folded in. The "recorded" message is the same for every cause. The rejected alternative was a variant for ErrFenced, which with the built-in Stores is reached only through an effect handler's error, where it misled. A refused checkpoint is retried only on a due step.

The other five dimensions were last confirmed before the last three commits, so they were rerun on b9e6074, along with a probe verify round on the final retry change. All came back with no blockers, so all eight dimensions are confirmed and the PR is ready. Their doc notes, plus one small fix (a deferred checkpoint's failure is hidden from its finishing caller for any context error, not only the one that ended its context), are in 0e9d2e7. CI is green on it.

@reka18 reka18 added confirmed:probes ACR: the probes lens's latest round found no blocker confirmed:api-compatibility ACR clean: api-compatibility confirmed:human-readability ACR clean: human-readability confirmed:missed-context ACR clean: missed-context confirmed:regression-safety ACR clean: regression-safety and removed wait:probes ACR: the probes lens is still open wait:api-compatibility ACR pending: api-compatibility wait:missed-context ACR pending: missed-context wait:human-readability ACR pending: human-readability wait:regression-safety ACR pending: regression-safety labels Nov 11, 2026
The README and overview gave a stale reason for stopping a healthy
Handle before handing its instance on: Drain now covers a step still
recording, and the reason is that a step can start after Drain returns.
The CHANGELOG now says a recorded step on a failed Handle returns
ErrHandleFailed, and that a retried deferred checkpoint's failure goes to
the step that tried it. Fire and Tick say how a checkpoint failure reaches
them. A deferred checkpoint is kept from its finishing caller for any
context error the Store reports, not only the one that ended that
caller's context.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
@reka18 reka18 added confirmed:replay-determinism ACR clean: replay-determinism ready All ACR dimensions confirmed - clear to merge and removed not-ready On every new PR until all review dimensions are confirmed wait:replay-determinism ACR pending: replay-determinism labels Nov 11, 2026
@reka18
reka18 merged commit 9ea0ded into main Nov 11, 2026
112 checks passed
@reka18
reka18 deleted the fix/durable-handle-lock-failed-append branch November 11, 2026 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

confirmed:api-compatibility ACR clean: api-compatibility confirmed:concurrency-discipline ACR clean: concurrency-discipline confirmed:correctness ACR clean: correctness confirmed:human-readability ACR clean: human-readability confirmed:missed-context ACR clean: missed-context confirmed:probes ACR: the probes lens's latest round found no blocker confirmed:regression-safety ACR clean: regression-safety confirmed:replay-determinism ACR clean: replay-determinism ready All ACR dimensions confirmed - clear to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants