Repository navigation
Commit 19955fb
fix(service-automation): settle the run row when a resume is refused because its flow or parked node is gone (#24003)
Fixes #23963
Clause-②: no
## What changed
`resumeInternal` has two never-dispatched exits for a suspension that is
real but can never continue: the run's flow is no longer registered, or
the node it parked on was edited away. The engine already classifies
both as the terminal `RUN_NOT_FOUND` class, and a transport answers them
404. Until now both exits returned without touching the run, so its
`sys_automation_run` row kept `status: 'paused'` through every later
resume, an uninstall and a cold restart. Retention never pruned it,
because retention's `onlyWhen` scope holds terminal statuses only.
Both exits now return through one new method,
`settleUnresumableSuspension`
(`packages/services/service-automation/src/engine.ts:9016`, called at
`:7799` and `:7803`). In the same call that answers the refusal, it does
three things:
1. It consumes the parked row through `claimAdvance` (`:9019`). That is
the compare-and-set the resume path already takes (`:8117`): the store
deletes the row only while it is still parked at the node this replica
read.
2. It releases the suspension through `forgetSuspendedRun(run, 'failed',
claimed)` (`:9038`), the one consumption choke point, so a `wait` node's
armed job is torn down too.
3. It records a `failed` history row whose `error` is the refusal's own
text, through `recordLog`.
The answer is byte-identical: same `code`, same `error` text. A second
resume finds nothing parked and answers `RUN_NOT_FOUND` (`No suspended
run 'ID'`). It writes nothing.
**Why `failed` and not `cancelled`.** `failSuspendedRun` sets the
precedent for an ancestor that can never be resumed. `cancelled`
(ADR-0044) records a deliberate act, and `restoreConsumedSuspension`
refuses it as one. Nobody decided to end this run. It can never
continue, and `error` is the column that says why. `failed` is already
in retention's `onlyWhen` list (`sys-automation-run.object.ts:67`), so
the row ages out by the existing rule. Retention is unchanged.
**Which refusals do NOT record anything.**
- **Before the host seals the boot** (`:9018`). The guard is
`nodeTypeVocabularySealed`, which `AutomationServicePlugin` sets at
`kernel:bootstrapped` (`plugin.ts:1592`), strictly after every flow bind
of the boot. Before that, an absent flow can be one that is not
registered yet. Measured on this branch: the plugin re-arms overdue wait
timers by calling `resume` inside `start()` (`plugin.ts:1649`), with the
engine unsealed. A flow its provider binds during `kernel:ready` is
absent at that moment. Without this guard, the plugin's own boot
recorded such a run `failed` a moment before its flow arrived (ablation
A below). Before the seal, the refusal is answered exactly as before and
nothing is recorded (AGENTS.md "Startup registry reads"). An embedded
host that never seals keeps the previous behaviour.
- **A claim another consumer won** (`lost`). A concurrent resume or
cancel already consumed the row and wrote its own outcome, so nothing is
written here. The stale snapshot is evicted, as on the resume path.
- **A claim the store cannot answer** (`unavailable`). The resume
answers `STORE_UNAVAILABLE` (503), as the resume path's claim does
(`:9029`). Answering `RUN_NOT_FOUND` there would close the matter for
the caller while the row could still read `paused`. A retry that finds
the run still parked settles it.
The history write is guarded the way the completion paths guard theirs.
The suspension is already consumed when it runs, so a throw out of
`recordLog` is logged at `error`, with the consequence and the fix
stated. It does not turn the refusal into a rejected promise.
## Tests
New file
`packages/services/service-automation/src/resume-gone-suspension-settle.test.ts`,
8 cases:
- **Full stack.** `ObjectKernel` + `ObjectQLPlugin` +
`AutomationServicePlugin` over a sqlite file, with the real
`ObjectStoreSuspendedRunStore` and a run parked on a real `screen` node.
Four cases:
- flow removed;
- parked node edited away;
- a second resume answers the same code and leaves the settled row
deep-equal;
- a cold restart onto a flow that disappeared while the process was down
settles on its first resume.
Each case reads `sys_automation_run` back: the paused row is gone when
`resume` returns, and the terminal row reads `failed` with the refusal
text in `error`. The status is asserted against
`SysAutomationRun.lifecycle.retention.onlyWhen.status.$in`, read off the
object, not restated.
- **Engine level.** A resume before the seal leaves the row parked, and
the run continues once its flow registers. A claim that throws answers
`STORE_UNAVAILABLE` with the row untouched, and the retry settles it. A
claim lost to a concurrent `cancelRun` leaves only the cancel's
`cancelled` record.
- **Plugin boot order.** Two process lifetimes over one sqlite file,
with a real `wait` node. A plugin composed after automation binds the
flow in its own `kernel:ready` handler. A positive control asserts that
the boot-time re-arm resume actually reached the flow-gone exit. After
the boot the row is still `paused`, and a resume completes it.
**Red at base** (`4153a7089f`, the pins against the unchanged engine): 6
failed, 2 passed. The first failure: `the paused row must not survive
the refusal: expected { …(32) } to be null`. The 2 that passed are the
two guards, which pin behaviour that must not change.
**Green:** the file 8/8. Package suite `pnpm --filter
@objectstack/service-automation exec vitest run --maxWorkers=2`: 189
files, 2533 tests passed. `pnpm --filter @objectstack/service-automation
typecheck` (tsc plus `check:test-typecheck`): exit 0. Both on the merged
head `0a1fd3171a`. The new file appears in both tsc programs
(`--listFiles`, `tsconfig.json` and `tsconfig.test.json`).
**Ablations.** The fix was committed first (`af6e1b4d2d`). Each mutation
went through `scripts/ablation-replace.mjs` (anchor hit 1 to 0, blob
changed) and was restored and proven: blob `a664529755` == HEAD, `git
diff HEAD` empty. The subject is imported from `src`, so no `dist` leg
applies.
| Ablation | Direction | Result |
|---|---|---|
| A. Seal guard deleted | Predicted the two guards red | 2 failed, 6
passed. Plugin boot order: `the run must still be parked after the boot:
expected undefined to be 'paused'` |
| B. Settle disabled (unconditional early return) | Predicted the six
settle pins red | 6 failed, 2 passed |
| C. `claimAdvance` replaced by an `unguarded` claim | Predicted the two
claim pins red | 2 failed, 6 passed. Lost claim: `expected [ [ 'failed',
…(1) ] ] to deeply equal [ [ 'cancelled', …(1) ] ]` |
## Gates
`node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` at `0a1fd3171a` derived 66 commands. All 66
were run, each exit captured before any pipe, and reconciled with
`--ran`: 65 at exit 0, and 1 NOT MEASURED. `pnpm
check:dual-build-cjs-loads` exited 3, `PREREQUISITE NOT MET`: it reads
built output for the whole tree and this worktree built only this
package's closure. That family is CI's reading. Other verdicts:
- `check:durability-log-level`: 48 durability-critical catch seams, all
loud.
- `check:nul-bytes`: OK.
- `check:doc-authoring`: clean.
- `check-adr-0087-registration`: no declared-breaking changeset.
- `check-empty-changeset`: one declaring changeset.
- `check:engine-double-contract`: OK.
## Acceptance notes
- **Remaining case, not filed.** An overdue `wait` run whose flow
disappeared while the process was down is not settled by the boot
re-arm, which runs before the seal. The row stays `paused`, as before,
until any later resume settles it. The re-arm ignores the resume's
answer (`rearmSuspendedWaitTimers` counts `rearmed++` on any
non-throwing result), so a refused overdue run is also counted in its
"Re-armed N" line. Moving the re-arm after `kernel:bootstrapped`, or
checking its result, would close both. That is `plugin.ts` and
`wait-node.ts`, outside this card's surface. Carrier: none.
- **Withdrawal-time settle.** The card's optional `withdrawFlow` settle
is not done. This change is the resume-time floor the card names.
- **Subflow children.** A parent parked at a `subflow` whose own flow is
gone is now settled `failed`. Its child run is its own run with its own
flow, and it stays resumable by its own id, as before.
- **No dogfood pin.** The HTTP status mapping (`RUN_NOT_FOUND` to 404,
`STORE_UNAVAILABLE` to 503,
`packages/runtime/src/domains/automation.ts`) is unchanged, and the
full-stack pins read the row on a real store.
- **Branch state.** `origin/main` was merged at `0a1fd3171a`, after PR
23954 (the `engine.ts` resolver-retirement hunks) landed. It merged
without conflict. Four later `main` commits touch `objectql`, `spec` and
`cli`, not `service-automation`.
---
_Generated by [Claude
Code](https://claude.ai/code/session_018VqJPGA1NDNSCZsnecsChP)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 92fd343 commit 19955fb
3 files changed
Lines changed: 613 additions & 2 deletions
File tree
- .changeset
- packages/services/service-automation/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7786 | 7786 | | |
7787 | 7787 | | |
7788 | 7788 | | |
| 7789 | + | |
| 7790 | + | |
| 7791 | + | |
| 7792 | + | |
| 7793 | + | |
| 7794 | + | |
| 7795 | + | |
| 7796 | + | |
7789 | 7797 | | |
7790 | 7798 | | |
7791 | | - | |
| 7799 | + | |
7792 | 7800 | | |
7793 | 7801 | | |
7794 | 7802 | | |
7795 | | - | |
| 7803 | + | |
| 7804 | + | |
| 7805 | + | |
| 7806 | + | |
7796 | 7807 | | |
7797 | 7808 | | |
7798 | 7809 | | |
| |||
8956 | 8967 | | |
8957 | 8968 | | |
8958 | 8969 | | |
| 8970 | + | |
| 8971 | + | |
| 8972 | + | |
| 8973 | + | |
| 8974 | + | |
| 8975 | + | |
| 8976 | + | |
| 8977 | + | |
| 8978 | + | |
| 8979 | + | |
| 8980 | + | |
| 8981 | + | |
| 8982 | + | |
| 8983 | + | |
| 8984 | + | |
| 8985 | + | |
| 8986 | + | |
| 8987 | + | |
| 8988 | + | |
| 8989 | + | |
| 8990 | + | |
| 8991 | + | |
| 8992 | + | |
| 8993 | + | |
| 8994 | + | |
| 8995 | + | |
| 8996 | + | |
| 8997 | + | |
| 8998 | + | |
| 8999 | + | |
| 9000 | + | |
| 9001 | + | |
| 9002 | + | |
| 9003 | + | |
| 9004 | + | |
| 9005 | + | |
| 9006 | + | |
| 9007 | + | |
| 9008 | + | |
| 9009 | + | |
| 9010 | + | |
| 9011 | + | |
| 9012 | + | |
| 9013 | + | |
| 9014 | + | |
| 9015 | + | |
| 9016 | + | |
| 9017 | + | |
| 9018 | + | |
| 9019 | + | |
| 9020 | + | |
| 9021 | + | |
| 9022 | + | |
| 9023 | + | |
| 9024 | + | |
| 9025 | + | |
| 9026 | + | |
| 9027 | + | |
| 9028 | + | |
| 9029 | + | |
| 9030 | + | |
| 9031 | + | |
| 9032 | + | |
| 9033 | + | |
| 9034 | + | |
| 9035 | + | |
| 9036 | + | |
| 9037 | + | |
| 9038 | + | |
| 9039 | + | |
| 9040 | + | |
| 9041 | + | |
| 9042 | + | |
| 9043 | + | |
| 9044 | + | |
| 9045 | + | |
| 9046 | + | |
| 9047 | + | |
| 9048 | + | |
| 9049 | + | |
| 9050 | + | |
| 9051 | + | |
| 9052 | + | |
| 9053 | + | |
| 9054 | + | |
| 9055 | + | |
| 9056 | + | |
| 9057 | + | |
| 9058 | + | |
| 9059 | + | |
| 9060 | + | |
| 9061 | + | |
| 9062 | + | |
| 9063 | + | |
| 9064 | + | |
| 9065 | + | |
| 9066 | + | |
| 9067 | + | |
| 9068 | + | |
| 9069 | + | |
| 9070 | + | |
| 9071 | + | |
8959 | 9072 | | |
8960 | 9073 | | |
8961 | 9074 | | |
| |||
0 commit comments