fix(orchestrator): bounded dispatch wait — advance_project never holds the MCP call for a whole worker run - #36
Open
keeganmccallum wants to merge 1 commit into
Conversation
…s the MCP call for a whole worker run The ACP dispatcher blocks on the worker's entire session, so advance_project held the orchestrator's MCP request for hours. MCP clients abort held requests (Prime Agent: 'Request was aborted' — the abort killed the caller's kernel cell), making a healthy dispatch look like 'nothing is running'. Fix: - StepResult.in_progress: advance returns promptly when workers are in flight; the next call reconciles their handoff files (workers write them directly). - .dispatched markers distinguish in-flight attempts from lost ones; the legacy 'running but no attempt file' crash-recovery only fires on missing or stale markers (attempt_stale_s, default 6h). - dispatch_wait_s (default 50s, ZENITH_DISPATCH_WAIT_S) bounds the wait — safely under MCP client timeouts. Tests: tests/test_bounded_dispatch.py (4) — prompt return under a slow worker, in_progress reconcile without fake failures, late-handoff application, lost and stale-marker paths. Full suite 217 passed, 7 skipped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The ACP dispatcher blocks on the worker's entire session, so
advance_projectheld the orchestrator's MCP request for the full worker run (hours). MCP clients abort held requests (Prime Agent:Request was aborted; the abort killed the caller's kernel cell), making a healthy dispatch look like 'nothing is running'.Fix
StepResult.in_progress:advance_projectreturns promptly when workers are in flight; the next call reconciles their handoff files (workers write them directly to the store — no state is lost by returning early)..dispatchedmarkers distinguish in-flight attempts from lost ones; the legacy 'running but no attempt file' crash-recovery path now only fires on missing or stale markers (attempt_stale_s, default 6h).dispatch_wait_s(default 50s,ZENITH_DISPATCH_WAIT_S) bounds the wait — safely under MCP client timeouts.Tests
tests/test_bounded_dispatch.py(4): prompt return under a slow worker, in_progress reconcile without fake failures, late-handoff application + marker cleanup, lost-attempt and stale-marker paths. Full suite: 217 passed, 7 skipped.