Repository navigation
fix(durable): lock the Handle and refuse steps after a failed Append - #243
Conversation
Findings from Temple's review of #233Temple reviewed #233, the PR this one replaces. These are his findings and where each one is handled. Line numbers are from #233's old head ( High
Medium
Low
A fresh review round runs on this PR before it changes, covering correctness, concurrency and replay determinism. The PRs above it will be rebased once it merges. |
Review of the rebased change (494c78a..2f8b57c)The change is rebased onto main after #232 merged, plus one fix commit (2f8b57c). All seven dimensions are confirmed, so the PR is ready. The reviewers (AI agents, one per dimension, each starting with a clean context) could only read code: they traced the code paths but couldn't run tests. I ran the tests and the mutation checks myself.
Fixed in 2f8b57c
Rejected: pre-existing on main, fixed higher in the stackThese come from code this PR doesn't change. Fixing them here would mean fixing more than the bottom PR of the stack.
Pre-existing, possibly not tracked yet
Non-blockers left as they are
VerificationOn
|
|
#245 passes the step's context to guards, actions, reducers and actor factories. Today only a service's call back into its own Handle is caught with |
Fire, Tick, RunService and DeliverToActor now hold a lock for the whole step, so a host's timer loop no longer loses acknowledged steps. A step that applied to the kernel but wasn't appended leaves the instance ahead of the journal, so the Handle refuses further steps with ErrHandleFailed until it's recovered. Relates to #214. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…rToActor RunService and DeliverToActor refuse steps after a failed Append and return an ended context's error, but only Fire and Tick were tested. New tests fail an Append under each and end the context before each, and check nothing ran or was journaled. The Tick context test now also checks that nothing was appended. The RunService and DeliverToActor docs said a failed step returns false; they return true with ErrHandleFailed once the service settled or the actor received the event. The docs now say so, name a context that ends during the Store write as a possible cause, and note that the latch relies on the Append running inline. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…w gaps An effect handler that called back into its own Handle deadlocked, because Fire held the step lock through dispatch. The lock now covers only the kernel step and the Record append. Effects dispatch after it's released, and a checkpoint waits until every step at or before it has dispatched, so compaction never drops an effect still being sent. A failed dispatch fails the Handle, so Recover sends the effect again. Code that has to run under the lock gets a marked context: a service that calls back in with it gets ErrReentrantStep instead of deadlocking. A step waiting for the lock stops when its context ends. Handle.View reads the instance under the lock. Also: - WithEventDataDecoder and DecodeAs decode a WithEventData payload as the caller's type, live and on replay. A payload that decodes as another type refuses the Fire. - StepReporter lets Recover number past every recorded step after a checkpoint from before checkpoints carried the next step. MemStore and FileStore implement it. - FileStore.Append encodes a Record once. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…ailing Review of the outbox change found gaps: - Fire released the step lock without defer, so a panic in host code under the lock left it held. It now releases however the step ends. - An effect handler that panicked left its step dispatching forever, which silently stopped every later checkpoint. A panic now fails the Handle with ErrEffectDispatch, like a returned error. - A call that fails the Handle returns only once no effect is still being sent, so a host that recovers right away can't send one twice. A step that held the lock while another goroutine failed the Handle leaves its effects for Recover instead of sending them. - refuse takes the context, so every step runs one check after taking the lock: the Handle hasn't failed, and the context hasn't ended. - A nil payload stays nil under a typed decoder, and ErrEffectDispatch wraps the handler's error. - Docs: the wait after a failure, handler failures failing the Handle, dispatch order for a step fired from a handler, StepReporter errors. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
A call that failed the Handle waited until no effect was still being sent. An effect handler that called back with a context other than its own, and failed the Handle, waited on its own dispatch forever, and so did every later caller. Calls that fail the Handle now return at once. Handle.Drain waits until no effect is still being sent; call it before Recover. Called from inside a step or an effect handler, it returns ErrReentrantStep instead of waiting. Docs: an action reaches the Handle by returning an effect; a guard, action, reducer or actor behavior that calls back without the step's context deadlocks. Fire's doc covers a failed checkpoint. The payload cap measures an error's JSON form, and a Store panic during dispatch is reported as a dispatch panic, not a handler's. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
Once a Handle has failed, no step can start sending effects, so Drain covers every one. On a healthy Handle a step still recording can start sending after Drain returns, so the docs say to stop calling it before moving its instance to a new Handle. Also: a typed-nil payload is refused as a type mismatch. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
560078a to
2c2eb92
Compare
…s Ctx With #245, a guard, action or reducer gets the step's context as its input's Ctx, so a call back into the Handle with it returns ErrReentrantStep instead of deadlocking. Say so in the docs, including that a call back without the step's context, or under middleware that passed down an unrelated one, still deadlocks. Cover Fire, an eventless transition, middleware, Tick, RunService and DeliverToActor in a test. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
2c2eb92 to
4626b47
Compare
govulncheck now fails every PR on GO-2026-6859, an HTTP/1 server crash in net/http that Go 1.27.4 fixes. It's reached through grpc's server in transport. Every go.mod and go.work now declares Go 1.27.4; CI reads the version from go.work. No dependency changes. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
Temple's Drain and checkpoint findingsBoth are fixed in b9e6074. Medium: Low: a deferred checkpoint used an unrelated caller's context. When the caller whose step finished last cancels, it's no longer told about a checkpoint it didn't make due, since its own step succeeded, and the next step writes that checkpoint. Before, the checkpoint was dropped and the error went to that caller. A retry the Store refuses as not advancing counts as written, because the first write landed. Any other checkpoint failure is still returned, and only the next step the policy makes due tries again, so other steps aren't told about a checkpoint they didn't make due. Five tests in Each fix is mutation-checked: with it removed, its test fails. How the review changedOur earlier rounds missed both bugs, so this round added two guards. The concurrency reviewer had to build a table of what else can be in flight against each changed method and whose context any deferred work uses. A new probe reviewer wrote and ran tests in a throwaway worktree to break the change. The probe found three more problems the read-only reviewers hadn't, each with a failing test, and all are fixed here:
Two non-blockers from the last probe round were Ray's call, and both are folded in. The "recorded" message is the same for every cause. The rejected alternative was a variant for The other five dimensions were last confirmed before the last three commits, so they were rerun on b9e6074, along with a probe verify round on the final retry change. All came back with no blockers, so all eight dimensions are confirmed and the PR is |
The README and overview gave a stale reason for stopping a healthy Handle before handing its instance on: Drain now covers a step still recording, and the reason is that a step can start after Drain returns. The CHANGELOG now says a recorded step on a failed Handle returns ErrHandleFailed, and that a retried deferred checkpoint's failure goes to the step that tried it. Fire and Tick say how a checkpoint failure reaches them. A deferred checkpoint is kept from its finishing caller for any context error the Store reports, not only the one that ended that caller's context. Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
What this change does
Two #214 fixes in the same code path, rebased onto main now that #232 and #245 have merged. This replaces #233, which GitHub couldn't reopen after a rebase. It also folds in the review of #233; my comment there lists each finding and where it landed.
Fire,Tick,RunServiceandDeliverToActorhold a lock while they apply to the kernel and record the step, so they're safe from different goroutines. A step waiting for the lock returns its context's error once the context ends, and changes nothing. This is the contract in my answer to question 10 on #214, which is a working answer, not a ruling.ErrEffectDispatchand leaves the effect forRecoverto send again.Append. A step that applied to the kernel but may not have been recorded fails the Handle. That call and every later one return the newErrHandleFailed, wrapping the cause. Call the newHandle.Drain, with a context of its own, thenRecovera new Handle once it returns nil.Drainwaits for a step still writing its Record, then for dispatch. A step recorded while another goroutine failed the Handle returnsErrHandleFailedsaying it was recorded.nextStepnow advances right after the append, so a failed checkpoint no longer makes the next step conflict.Ctxfrom feat(state): pass the step's context to callbacks; skip CI on drafts #245) gets the newErrReentrantStepat once. A call without that context still deadlocks; the docs say so, and say to return an effect instead.refuse(ctx), returnsErrHandleFailedfor a failed Handle andctx.Err()for an ended context.Viewreads the instance under the lock.Instance()still returns it outside.MemStoreandFileStoreimplement the newStepReporter, soRecovernumbers the next step past every recorded one. A checkpoint from before fix(durable): carry the next step through a checkpoint #232 followed by a tick barrier no longer makes everyAppendconflict.Assigndecoded, live as on replay.WithEventDataDecoderdecodes a payload as the caller's type, chosen per event, withDecodeAs[T]. A payload that decodes as a different type than the caller passed refuses theFirewithErrEventDataEncoding, rather than reachingAssignas a map.FileStore.Appendclones the record once.go.modandgo.workmoves to Go 1.27.4, which fixes GO-2026-6859, an HTTP/1 server crash innet/httpthat govulncheck now fails every PR on. No dependency changes.Please check these:
Appendstays inline, on the session's single owner goroutine. If it moves off that goroutine, the lock scope and this failure handling change.state.ServiceRunnerruns and settles in one call, so I couldn't narrow it here.ErrFenced, a newer owner holds the instance, soRecoverwon't help. The docs say to stop.Append, a service whose result wasn't recorded is in flight again on the recovered Handle, soRunServiceruns it a second time. AnInvokeActoractor restarts from its initial state, and aSpawnactor doesn't return.Known and left open:
Storethat panics mid-step releases the lock but skips the failure latch, if the host recovers the panic.Runner.Firecalls on one id aren't serialized. The doc says so.Fireleaves a Trace only in the live instance, so a later snapshot can differ byte for byte.Startdoesn't pass its context to the callbacks that settle the initial state, andcluster.WithActorBehaviorshas no form that takes a context. Neither can reach a Handle's lock today.Why
A host's timer loop racing
Firelost acknowledged steps, and a failedAppendleft the live instance ahead of the journal.Relates to: #214, #245
Evidence
On
GOTOOLCHAIN=go1.27.4:go test -race ./state/... ./durable/... ./cluster/...go test ./...inexamples/dispatchande2egolangci-lintondurablegovulncheckin each module CI scansgo test ./...ingenandtransportThe new tests in
handle_lock_test.go,handle_failure_test.go,runner_test.goandevent_data_test.gocover the timer loop racingFire, a failedAppendon each step, an ended context, a deadline in the lock queue, a panic under the lock, dispatch outside the lock and a checkpoint waiting for it, a failed or panicking effect handler,Drain,View, a service, guard, action or reducer calling back in,StepReporterrecovery and event data decoding. Each fails with the code it covers removed. The timer-loop test is reliable under-race; a plain run can pass by luck.Checklist
git commit -s) per the DCOtype: subject)mage checkpasses locally (lint + race tests + govulncheck): not run as a whole; the parts above were run individually