Skip to content

Fix reconcile lock expiry during aggregation - #536

Merged
chubes4 merged 6 commits into
mainfrom
fix/532-reconcile-lock-fencing
Aug 26, 2026
Merged

Fix reconcile lock expiry during aggregation#536
chubes4 merged 6 commits into
mainfrom
fix/532-reconcile-lock-fencing

Conversation

@chubes4

@chubes4 chubes4 commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Summary

  • closes fix: prevent reconcile lock expiry during aggregation #532 by limiting the option-backed reconcile lock to short recorder transitions
  • adds a dedicated, unique Action Scheduler aggregate continuation between reconcile-only receipts and resume
  • preserves main's admission-token/store-backend contracts and failed branch/reconcile recovery lifecycle
  • provides bounded Action Scheduler claim headroom and one run-scoped cleanup surface across branch, reconcile, aggregate, and resume hooks

Root cause

Reconciliation held a fixed 60-second option lock while executing arbitrary aggregate workflow steps. Agent and ability-backed aggregation can legitimately exceed that lease, allowing another reconciler to reclaim the lock while the original process continued.

Durable continuation

For Action Scheduler-owned runs, the all-terminal reconcile transition enqueues one unique wp_agent_workflow_run_aggregate action and persists its generation-bound owner.

  • queued: the durable aggregate action exists; duplicate reconcile delivery is a no-op
  • running: the claimed action crossed the durable effect-start boundary; no duplicate action may rerun aggregation
  • committed: aggregate output or an honest uncertain-outcome failure is durable; only resume remains

The action transitions queued to running under the short reconcile lock, aggregates outside the lock, then reacquires the lock and commits only if its owner token and suspension generation still match.

Failure lifecycle

The canonical Action Scheduler failure registrations preserve main's branch/reconcile receipt recovery and additionally route aggregate actions by hook:

  • failed queued aggregate: safely re-enqueue because effects did not begin
  • failed running aggregate: persist workflow_parallel_aggregation_outcome_uncertain; never rerun effects
  • failed after committed: enqueue/resume from durable output
  • recorder failure during failed-action recovery: enqueue the aggregate hook in recovery mode until the terminal transition succeeds

There is no aggregate claim timeout and no reconcile hot retry while healthy aggregation is active.

Main integration

Merged origin/main at 1d1b9f06 with a normal merge commit (e914e65), preserving PRs #535 and #534:

  • atomic partial-admission tokens, compensation, and backend provenance
  • persisted reconcile-only receipts and opaque continuation state
  • failed branch/reconcile recovery without repeating branch effects
  • custom receipt stores, legacy payload compatibility, and deterministic cleanup
  • single-winner terminal completion and run-scoped cleanup funnel

The scoped drain/cleanup hooks are now ordered as branch, reconcile, aggregate, and resume. Concurrency headroom is additive and bounded for all four; batch size is pinned to one while branch or aggregate effects are active.

Verification

Focused combined matrix:

  • php tests/workflow-as-branch-smoke.php (136 assertions)
  • php tests/workflow-branch-concurrency-gate-smoke.php (44 assertions)
  • php tests/workflow-async-branch-payload-smoke.php (40 assertions)
  • php tests/workflow-scoped-drain-smoke.php
  • php tests/workflow-request-controller-smoke.php
  • php tests/workflow-reconcile-race-smoke.php
  • php tests/workflow-reconcile-branch-authorization-smoke.php
  • php tests/workflow-parallel-async-smoke.php

Project gates:

  • composer test
  • composer phpstan
  • git diff --check
  • git diff --cached --check

Coverage includes partial admission, branch reconcile contention, failed branch/reconcile recovery, healthy long aggregation, aggregate crash lifecycle, committed-before-resume recovery, duplicate delivery, custom stores, legacy/tokenless payloads, bounded four-hook headroom, and terminal completion/cleanup single-winner behavior.

AI assistance

Implemented and tested with AI assistance. The final integration was reviewed against current main, Action Scheduler claim/failure semantics, pluggable store ownership, and the complete project test/static-analysis gates.

…k-fencing

# Conflicts:
#	src/Workflows/class-wp-agent-workflow-scoped-drain.php
#	src/Workflows/register-workflow-branch-executor.php
#	stubs/action-scheduler-classes.php
#	tests/workflow-as-branch-smoke.php
#	tests/workflow-request-controller-smoke.php
#	tests/workflow-scoped-drain-smoke.php
@chubes4
chubes4 merged commit fba3f59 into main Aug 26, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: prevent reconcile lock expiry during aggregation

1 participant