Skip to content

Commit f86d733

Browse files
committed
fix(execution): have a workflow-group cancellation report the writes it made
Three review findings landed on the same reporting logic, each a different face of one cause: the caller could not see what the group transaction had written, so it inferred. It inferred from an entry snapshot, then from the returned kind, and the remaining blind spot was the kind that covers two different transactions — a repair that terminalizes an active log, and a genuine no-op — which left a cancel that wrote nothing still claiming a durable write when it lost a race. The transaction now reports both writes it can make, each read from that statement's own returning row and recorded immediately before the throw that already depended on it, so the report cannot drift from the write. The caller derives its outcome from those rather than from the kind, and the kind is back to naming the situation instead of standing in for the work. The group path can now always answer whether it wrote. The only remaining unknown is the direct claim when its update throws or is never attempted, which genuinely has no row count to report.
1 parent c4e6abb commit f86d733

4 files changed

Lines changed: 172 additions & 44 deletions

File tree

apps/sim/lib/execution/cancel-workflow-execution.test.ts

Lines changed: 95 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,16 @@ vi.mock('@/lib/workflows/executor/human-in-the-loop-manager', () => ({
101101

102102
import { cancelWorkflowExecution } from '@/lib/execution/cancel-workflow-execution'
103103

104+
/**
105+
* The durable writes a workflow-group transition reports back. The transaction
106+
* updates the workflow log only, the cell sidecar only, or both, so a single
107+
* `kind` cannot answer whether this request wrote anything.
108+
*/
109+
const NO_WRITES = { workflowLogTerminalized: false, sidecarCancelled: false } as const
110+
const LOG_WRITE = { workflowLogTerminalized: true, sidecarCancelled: false } as const
111+
const SIDECAR_WRITE = { workflowLogTerminalized: false, sidecarCancelled: true } as const
112+
const BOTH_WRITES = { workflowLogTerminalized: true, sidecarCancelled: true } as const
113+
104114
const INPUT = {
105115
executionId: 'execution-1',
106116
workflowId: 'workflow-1',
@@ -303,6 +313,7 @@ describe('cancelWorkflowExecution', () => {
303313
tableId: 'table-1',
304314
rowId: 'row-1',
305315
groupId: 'group-1',
316+
writes: BOTH_WRITES,
306317
}
307318
mockCancelWorkflowGroupExecution.mockResolvedValue(cancelled)
308319

@@ -337,6 +348,7 @@ describe('cancelWorkflowExecution', () => {
337348
tableId: 'table-1',
338349
rowId: 'row-1',
339350
groupId: 'group-1',
351+
writes: SIDECAR_WRITE,
340352
}
341353
mockCancelWorkflowGroupExecution.mockResolvedValue(cancelled)
342354

@@ -356,7 +368,10 @@ describe('cancelWorkflowExecution', () => {
356368
workflowGroupWorkspaceId: 'workspace-1',
357369
priorStatus: 'running',
358370
})
359-
mockCancelWorkflowGroupExecution.mockResolvedValue({ kind: 'cancelled_without_sidecar' })
371+
mockCancelWorkflowGroupExecution.mockResolvedValue({
372+
kind: 'cancelled_without_sidecar',
373+
writes: LOG_WRITE,
374+
})
360375

361376
const result = await cancelWorkflowExecution(INPUT)
362377

@@ -380,6 +395,71 @@ describe('cancelWorkflowExecution', () => {
380395
tableId: 'table-1',
381396
rowId: 'row-1',
382397
groupId: 'group-1',
398+
writes: NO_WRITES,
399+
})
400+
401+
const result = await cancelWorkflowExecution(INPUT)
402+
403+
expect(result).toMatchObject({
404+
success: true,
405+
durablyRecorded: false,
406+
reason: 'already_cancelled',
407+
})
408+
})
409+
410+
/**
411+
* The same `already_cancelled` kind covers a transition that left the sidecar
412+
* alone but still terminalized an active workflow log. That log write is
413+
* durable, so the outcome must stay `recorded` and must not re-read a state
414+
* this request itself wrote.
415+
*/
416+
it('reports a durable write when a group run only repaired its workflow log', async () => {
417+
mockResolveWorkflowExecutionOwnership.mockResolvedValue({
418+
belongsToWorkflow: true,
419+
workflowGroupWorkspaceId: 'workspace-1',
420+
priorStatus: 'running',
421+
})
422+
mockCancelWorkflowGroupExecution.mockResolvedValue({
423+
kind: 'already_cancelled',
424+
tableId: 'table-1',
425+
rowId: 'row-1',
426+
groupId: 'group-1',
427+
writes: LOG_WRITE,
428+
})
429+
430+
const result = await cancelWorkflowExecution(INPUT)
431+
432+
expect(result).toMatchObject({ success: true, durablyRecorded: true, reason: 'recorded' })
433+
expect(mockResolveWorkflowExecutionOwnership).toHaveBeenCalledOnce()
434+
})
435+
436+
/**
437+
* The lost race the sidecar-bearing kind used to hide: a concurrent cancel
438+
* terminalized both records between the entry snapshot and this transaction,
439+
* which then found the sidecar already `cancelled` and the log already
440+
* `cancelled` and wrote nothing. A non-terminal entry snapshot cannot catch
441+
* that, so the transition's own report of having written nothing is what
442+
* forces the re-read — otherwise the request would claim a durable write and
443+
* fire the v2 cancel analytics gate on a no-op.
444+
*/
445+
it('reports a group run that lost the race with its sidecar already cancelled as a no-op', async () => {
446+
mockResolveWorkflowExecutionOwnership
447+
.mockResolvedValueOnce({
448+
belongsToWorkflow: true,
449+
workflowGroupWorkspaceId: 'workspace-1',
450+
priorStatus: 'running',
451+
})
452+
.mockResolvedValueOnce({
453+
belongsToWorkflow: true,
454+
workflowGroupWorkspaceId: 'workspace-1',
455+
priorStatus: 'cancelled',
456+
})
457+
mockCancelWorkflowGroupExecution.mockResolvedValue({
458+
kind: 'already_cancelled',
459+
tableId: 'table-1',
460+
rowId: 'row-1',
461+
groupId: 'group-1',
462+
writes: NO_WRITES,
383463
})
384464

385465
const result = await cancelWorkflowExecution(INPUT)
@@ -389,6 +469,7 @@ describe('cancelWorkflowExecution', () => {
389469
durablyRecorded: false,
390470
reason: 'already_cancelled',
391471
})
472+
expect(mockResolveWorkflowExecutionOwnership).toHaveBeenCalledTimes(2)
392473
})
393474

394475
/**
@@ -410,6 +491,7 @@ describe('cancelWorkflowExecution', () => {
410491
})
411492
mockCancelWorkflowGroupExecution.mockResolvedValue({
412493
kind: 'already_cancelled_without_sidecar',
494+
writes: NO_WRITES,
413495
})
414496

415497
const result = await cancelWorkflowExecution(INPUT)
@@ -423,8 +505,14 @@ describe('cancelWorkflowExecution', () => {
423505
})
424506

425507
it.each([
426-
[{ kind: 'conflict' as const, status: 'completed' }, 'cannot be cancelled while completed'],
427-
[{ kind: 'not_workflow_group' as const }, 'no longer the active table execution'],
508+
[
509+
{ kind: 'conflict' as const, status: 'completed', writes: NO_WRITES },
510+
'cannot be cancelled while completed',
511+
],
512+
[
513+
{ kind: 'not_workflow_group' as const, writes: NO_WRITES },
514+
'no longer the active table execution',
515+
],
428516
])(
429517
'releases the reservation before reporting a refused workflow-group cell claim as a conflict',
430518
async (outcome, message) => {
@@ -444,8 +532,8 @@ describe('cancelWorkflowExecution', () => {
444532
)
445533

446534
it.each([
447-
[{ kind: 'conflict' as const, status: 'completed' }],
448-
[{ kind: 'not_workflow_group' as const }],
535+
[{ kind: 'conflict' as const, status: 'completed', writes: NO_WRITES }],
536+
[{ kind: 'not_workflow_group' as const, writes: NO_WRITES }],
449537
])(
450538
'keeps the reservation held when a refused claim follows a paused cancellation',
451539
async (outcome) => {
@@ -462,8 +550,8 @@ describe('cancelWorkflowExecution', () => {
462550
)
463551

464552
it.each([
465-
[{ kind: 'conflict' as const, status: 'completed' }],
466-
[{ kind: 'not_workflow_group' as const }],
553+
[{ kind: 'conflict' as const, status: 'completed', writes: NO_WRITES }],
554+
[{ kind: 'not_workflow_group' as const, writes: NO_WRITES }],
467555
])(
468556
'keeps the reservation held when a refused claim follows a failed cancellation',
469557
async (outcome) => {

apps/sim/lib/execution/cancel-workflow-execution.ts

Lines changed: 15 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ import {
1818
cancelWorkflowGroupExecution,
1919
type PublishableWorkflowGroupCancellation,
2020
publishWorkflowGroupCancellationEvent,
21-
type WorkflowGroupExecutionCancellationResult,
21+
type WorkflowGroupCancellationWrites,
2222
} from '@/lib/table/workflow-group-cancellation'
2323
import { WORKFLOW_EXECUTION_JOB_ID_PREFIX } from '@/lib/workflows/executor/execution-job-ids'
2424
import { resolveWorkflowExecutionOwnership } from '@/lib/workflows/executor/execution-queries'
@@ -85,28 +85,24 @@ function toTerminalExecutionStatus(
8585
* path that can terminalize the run — the direct log claim and the
8686
* workflow-group transition — answers in this one vocabulary, so the report can
8787
* ask a single question: did this request durably write?
88+
*
89+
* Only the direct claim ever answers `unknown`, and only when it could not run
90+
* or its statement failed. The workflow-group transition always knows: it
91+
* reports the writes it performed.
8892
*/
8993
type TerminalWriteOutcome = 'applied' | 'no_row' | 'unknown'
9094

9195
/**
92-
* What a returned workflow-group transition durably wrote. `cancelled` claims
93-
* the cell sidecar and `cancelled_without_sidecar` terminalizes the workflow log
94-
* itself, so both are writes this request performed — including the reconciling
95-
* cancel of a sidecar left in `error` behind an already-`cancelled` log.
96-
* `already_cancelled_without_sidecar` found the log already `cancelled` and
97-
* touched nothing. `already_cancelled` left the sidecar alone but may still have
98-
* terminalized an active workflow log, which the result does not distinguish, so
99-
* it cannot claim either way. `conflict` and `not_workflow_group` never reach
100-
* the report — both throw above — and are mapped only to keep this map total.
96+
* Reads a workflow-group transition's durability off the writes it reported
97+
* rather than off its `kind`. Terminalizing the workflow log and cancelling the
98+
* cell sidecar are each a durable write this request performed, and a single
99+
* `kind` covers both a transition that did one of them and one that did
100+
* neither: `already_cancelled` leaves a sidecar that was already `cancelled`
101+
* alone, but may still have terminalized an active workflow log.
101102
*/
102-
const WORKFLOW_GROUP_TERMINAL_WRITES = {
103-
cancelled: 'applied',
104-
cancelled_without_sidecar: 'applied',
105-
already_cancelled: 'unknown',
106-
already_cancelled_without_sidecar: 'no_row',
107-
conflict: 'unknown',
108-
not_workflow_group: 'unknown',
109-
} as const satisfies Record<WorkflowGroupExecutionCancellationResult['kind'], TerminalWriteOutcome>
103+
function toTerminalWriteOutcome(writes: WorkflowGroupCancellationWrites): TerminalWriteOutcome {
104+
return writes.workflowLogTerminalized || writes.sidecarCancelled ? 'applied' : 'no_row'
105+
}
110106

111107
/**
112108
* Names the terminal state the cancel could not move, or `null` when it did
@@ -474,7 +470,7 @@ export async function cancelWorkflowExecution(
474470
*/
475471
let terminalWrite: TerminalWriteOutcome = 'unknown'
476472
if (groupCancellation !== null) {
477-
terminalWrite = WORKFLOW_GROUP_TERMINAL_WRITES[groupCancellation.kind]
473+
terminalWrite = toTerminalWriteOutcome(groupCancellation.writes)
478474
} else if (
479475
(cancellation.durablyRecorded || queuedJobCancelled || locallyAborted) &&
480476
!pausedCancelled

apps/sim/lib/table/workflow-group-cancellation.test.ts

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,8 @@ const OPTIONS = {
2323
executionId: 'execution-1',
2424
}
2525

26+
const NO_WRITES = { workflowLogTerminalized: false, sidecarCancelled: false } as const
27+
2628
const ACTIVE_TARGET = {
2729
tableId: 'table-1',
2830
rowId: 'row-1',
@@ -64,6 +66,7 @@ describe('cancelWorkflowGroupExecution', () => {
6466
rowId: 'row-1',
6567
groupId: 'group-1',
6668
blockErrors: { 'block-1': 'Provider failed' },
69+
writes: { workflowLogTerminalized: true, sidecarCancelled: true },
6770
})
6871

6972
expect(dbChainMockFns.transaction).toHaveBeenCalledOnce()
@@ -165,6 +168,7 @@ describe('cancelWorkflowGroupExecution', () => {
165168

166169
await expect(cancelWorkflowGroupExecution(OPTIONS)).resolves.toEqual({
167170
kind: 'not_workflow_group',
171+
writes: NO_WRITES,
168172
})
169173

170174
expect(dbChainMockFns.update).not.toHaveBeenCalled()
@@ -178,6 +182,7 @@ describe('cancelWorkflowGroupExecution', () => {
178182

179183
await expect(cancelWorkflowGroupExecution(OPTIONS)).resolves.toEqual({
180184
kind: 'not_workflow_group',
185+
writes: NO_WRITES,
181186
})
182187

183188
expect(dbChainMockFns.update).not.toHaveBeenCalled()
@@ -197,6 +202,7 @@ describe('cancelWorkflowGroupExecution', () => {
197202

198203
await expect(cancelWorkflowGroupExecution(OPTIONS)).resolves.toEqual({
199204
kind: 'not_workflow_group',
205+
writes: NO_WRITES,
200206
})
201207

202208
expect(dbChainMockFns.update).not.toHaveBeenCalled()
@@ -216,6 +222,7 @@ describe('cancelWorkflowGroupExecution', () => {
216222

217223
await expect(cancelWorkflowGroupExecution(OPTIONS)).resolves.toEqual({
218224
kind: 'cancelled_without_sidecar',
225+
writes: { workflowLogTerminalized: true, sidecarCancelled: false },
219226
})
220227

221228
expect(dbChainMockFns.update).toHaveBeenCalledOnce()
@@ -249,6 +256,7 @@ describe('cancelWorkflowGroupExecution', () => {
249256

250257
await expect(cancelWorkflowGroupExecution(OPTIONS)).resolves.toEqual({
251258
kind: 'already_cancelled_without_sidecar',
259+
writes: NO_WRITES,
252260
})
253261

254262
expect(dbChainMockFns.update).not.toHaveBeenCalled()
@@ -269,6 +277,7 @@ describe('cancelWorkflowGroupExecution', () => {
269277

270278
await expect(cancelWorkflowGroupExecution(OPTIONS)).resolves.toEqual({
271279
kind: 'not_workflow_group',
280+
writes: NO_WRITES,
272281
})
273282

274283
expect(dbChainMockFns.update).not.toHaveBeenCalled()
@@ -288,6 +297,7 @@ describe('cancelWorkflowGroupExecution', () => {
288297
await expect(cancelWorkflowGroupExecution(OPTIONS)).resolves.toEqual({
289298
kind: 'conflict',
290299
status: 'completed',
300+
writes: NO_WRITES,
291301
})
292302

293303
expect(dbChainMockFns.update).not.toHaveBeenCalled()
@@ -318,6 +328,7 @@ describe('cancelWorkflowGroupExecution', () => {
318328
await expect(cancelWorkflowGroupExecution(OPTIONS)).resolves.toEqual({
319329
kind: 'conflict',
320330
status: 'completed',
331+
writes: NO_WRITES,
321332
})
322333

323334
expect(dbChainMockFns.update).not.toHaveBeenCalled()
@@ -335,6 +346,7 @@ describe('cancelWorkflowGroupExecution', () => {
335346
rowId: 'row-1',
336347
groupId: 'group-1',
337348
blockErrors: { 'block-1': 'Provider failed' },
349+
writes: NO_WRITES,
338350
})
339351

340352
expect(dbChainMockFns.update).not.toHaveBeenCalled()
@@ -353,6 +365,7 @@ describe('cancelWorkflowGroupExecution', () => {
353365
rowId: 'row-1',
354366
groupId: 'group-1',
355367
blockErrors: { 'block-1': 'Provider failed' },
368+
writes: { workflowLogTerminalized: false, sidecarCancelled: true },
356369
})
357370

358371
expect(dbChainMockFns.update).toHaveBeenCalledOnce()
@@ -371,6 +384,7 @@ describe('cancelWorkflowGroupExecution', () => {
371384
rowId: 'row-1',
372385
groupId: 'group-1',
373386
blockErrors: { 'block-1': 'Provider failed' },
387+
writes: { workflowLogTerminalized: false, sidecarCancelled: true },
374388
})
375389

376390
const sidecarUpdateValues = collectConditionValues(dbChainMockFns.where.mock.calls[2]?.[0])
@@ -395,6 +409,7 @@ describe('cancelWorkflowGroupExecution', () => {
395409
await expect(cancelWorkflowGroupExecution(OPTIONS)).resolves.toEqual({
396410
kind: 'conflict',
397411
status: 'error',
412+
writes: NO_WRITES,
398413
})
399414

400415
expect(dbChainMockFns.update).not.toHaveBeenCalled()
@@ -413,6 +428,7 @@ describe('cancelWorkflowGroupExecution', () => {
413428
rowId: 'row-1',
414429
groupId: 'group-1',
415430
blockErrors: { 'block-1': 'Provider failed' },
431+
writes: { workflowLogTerminalized: true, sidecarCancelled: false },
416432
})
417433

418434
expect(dbChainMockFns.update).toHaveBeenCalledOnce()
@@ -429,6 +445,7 @@ describe('cancelWorkflowGroupExecution', () => {
429445
await expect(cancelWorkflowGroupExecution(OPTIONS)).resolves.toEqual({
430446
kind: 'conflict',
431447
status,
448+
writes: NO_WRITES,
432449
})
433450

434451
expect(dbChainMockFns.update).not.toHaveBeenCalled()

0 commit comments

Comments
 (0)