Skip to content

Make generic run-control mutations atomic (workflow cancel/finish race) - #492

Merged
chubes4 merged 8 commits into
Automattic:mainfrom
lezama:fix/run-control-atomic-mutations
Aug 26, 2026
Merged

Make generic run-control mutations atomic (workflow cancel/finish race)#492
chubes4 merged 8 commits into
Automattic:mainfrom
lezama:fix/run-control-atomic-mutations

Conversation

@lezama

@lezama lezama commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Make generic run-control lifecycle mutations atomic and route every terminal producer through its authoritative stored winner.

Changes

  • Atomic site/workspace lifecycle mutations with typed retryable failure propagation.
  • Monotonic terminal state, deterministic cancellation ordering, and legacy contradiction healing.
  • Exact-once runtime-package run admission with idempotent terminal replay and active duplicate rejection.
  • Canonical skipped support in generic run-control so skipped runtime-package results remain terminal and replay without handler re-execution.
  • Centralized runtime-package projection for success, skipped, failure, cancellation, and every error exit.
  • Centralized workflow terminal projection before recorder writes, returns, and completion hooks.
  • Generic budget_exceeded, stalled, and interrupted winners map to honest failed workflow results with stable error codes.
  • Recorder-start failure now terminalizes run-control and publishes completion once even when persistence cannot start.
  • Resume failures update an available recorder, terminalize run-control, publish once, clear suspension state, and replay idempotently instead of leaving a suspended/running zombie.
  • Async reconcile-recovery failures use the same authoritative workflow projection.

Testing

  • tests/run-control-atomic-mutations-smoke.php: 56 assertions pass.
  • tests/runtime-package-run-contract-smoke.php: 71 assertions pass, including skipped storage/replay and all error-exit races.
  • tests/workflow-runner-smoke.php: 100 assertions pass, including generic terminal winners, recorder-start completion, and checked resume-failure idempotency.
  • tests/workflow-as-branch-smoke.php: 141 assertions pass, including reconcile-recovery winner parity.
  • composer test: pass.
  • composer phpstan: no errors.
  • git diff --check: pass.

Current origin/main remains d269bc5; no additional mainline merge was needed and no history was rewritten.

lezama and others added 3 commits August 6, 2026 17:42
The generic lifecycle mutations on WP_Agent_Run_Control (start_run,
save_run, finish_run, request_cancel) performed a non-atomic
state() -> mutate -> save_state() read-modify-write with no locking.
These generic methods back workflow run-control
(WP_Agent_Workflow_Runner, WP_Agent_Workflow_Request_Controller) and
runtime-package runs, so a workflow request_cancel
(register-agents-workflow-abilities.php) could race the runner's
finish_run (class-wp-agent-workflow-runner.php) on the same store and
run_id: both read the same base state and the second save silently
overwrites the first, losing the cancel or the terminal status.

Route all four mutations through the store's atomic read-modify-write
path (mutate_state / mutate_workspace_state) via a new private
mutate_run_state() helper, closing the lost-update window. This mirrors
commit 8980da5, which hardened the chat layer
(WP_Agent_Chat_Run_Control) to the atomic path but left these generic
methods non-atomic. Stores without atomic support, and environments
without a database connection (pure-PHP execution before WordPress
boots), keep their historical non-atomic behavior, matching the chat
layer's wpdb-availability guard.

Adds tests/run-control-atomic-mutations-smoke.php, which uses a store
spy to assert the four mutations take the atomic mutate path (not plain
save_state) and that an interleaved concurrent write is not lost. The
test fails without the fix and passes with it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@chubes4
chubes4 marked this pull request as ready for review August 26, 2026 21:56
@chubes4
chubes4 merged commit 3d29c30 into Automattic:main Aug 26, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants