Repository navigation
Description
Observed Behavior
If I stop iterating a workflow stream in the middle of a superstep and start a new run on the same Workflow before the dropped stream's async generator has been finalized, the late finalizer discards the new run's pending state writes. The new run still finishes IDLE with its outputs, so the lost write is silent: nothing is raised or logged.
Expected Behavior
Cleanup from the abandoned run should only touch its own superstep. The second run's ctx.set_state("b_key", ...) should be committed and visible in the workflow state after it finishes.
Steps to Reproduce
- Build a workflow whose executor stages a state write and then blocks (run A).
- Iterate run A's stream until
executor_invoked, then drop the stream and callgc.collect(). The generator'saclose()is now scheduled on the loop but has not run. - Start run B on the same workflow. Its handler stages
b_keyand waits. - Let the loop run A's finalizer, then let B finish.
b_keyis missing from the state, although B completed normally.
Minimal Reproduction
"""A dropped workflow stream, finalized late, discards the next run's state writes.
Run A stages a state write and blocks inside its superstep. The caller stops
iterating A's stream and starts run B on the same Workflow before A's async
generator finalizer has run. B stages its own write. When A's finalizer runs,
it discards the pending writes, which now belong to B. B finishes IDLE with its
output and its write is gone.
"""
import asyncio
import gc
from dataclasses import dataclass
from agent_framework import Executor, WorkflowBuilder, WorkflowContext, handler
@dataclass
class Msg:
run: str
a_staged = asyncio.Event()
b_staged = asyncio.Event()
release_b = asyncio.Event()
class Writer(Executor):
@handler
async def handle(self, message: Msg, ctx: WorkflowContext[Msg, str]) -> None:
if message.run == "A":
ctx.set_state("a_key", "from-run-A")
a_staged.set()
await asyncio.Event().wait() # never finishes; A is abandoned
else:
ctx.set_state("b_key", "from-run-B")
b_staged.set()
await release_b.wait()
await ctx.yield_output("b-done")
async def main() -> None:
workflow = WorkflowBuilder(start_executor=Writer(id="writer")).build()
# Run A: iterate until the executor is invoked, then drop the stream.
stream_a = workflow.run(Msg("A"), stream=True)
it_a = stream_a.__aiter__()
while True:
event = await it_a.__anext__()
if event.type == "executor_invoked":
break
await a_staged.wait()
del stream_a, it_a, event
gc.collect() # A's finalizer is now scheduled on the loop, not yet run
# Run B on the same workflow.
async def consume_b():
outputs, final_state = [], None
async for ev in workflow.run(Msg("B"), stream=True):
if ev.type == "output":
outputs.append(ev.data)
elif ev.type == "status":
final_state = ev.state
return outputs, final_state
task_b = asyncio.create_task(consume_b())
await asyncio.wait_for(b_staged.wait(), 5)
# Let A's deferred finalizer run, then let B finish.
for _ in range(20):
await asyncio.sleep(0.01)
release_b.set()
outputs, final_state = await asyncio.wait_for(task_b, 5)
b_key = workflow._runner.state.export_state().get("b_key")
print(f"run B outputs: {outputs}")
print(f"run B final state: {final_state}")
print(f"observed b_key after run B: {b_key!r}")
print("expected b_key after run B: 'from-run-B'")
print("BUG REPRODUCED" if b_key != "from-run-B" else "not reproduced")
asyncio.run(main())Output:
run B outputs: ['b-done']
run B final state: WorkflowRunState.IDLE
observed b_key after run B: None
expected b_key after run B: 'from-run-B'
BUG REPRODUCED
With State.discard patched to a no-op the same script prints b_key as 'from-run-B', so the write is lost in the discard.
Error Messages and Stack Traces
None. No exception and no log output.
Package Versions
agent-framework-core: 1.22.0 (python-1.22.0)
Python Version
Python 3.12.3
Operating System
Linux
Regression
Yes
Additional Context
The cause is the finally: if not committed: self._state.discard() around each superstep in RunnerImpl.run_until_convergence (python/packages/core/agent_framework/_workflows/_runner.py lines 127-171 at 44fe223, the discard at 169-171), added in #8819 for #7859. The State object is created once in Workflow.__init__ (_workflow.py line 395) and shared by every run on that instance, so when run A's generator is closed late its discard() clears whatever is pending at that moment, which is run B's write. python-1.21.0 has no discard in the runner, and with the discard disabled the same script commits run B's write.
Workflow._run_core already handles this ordering for its own cleanup: the comment at _workflow.py 921-928 describes exactly this case (a stream dropped after partial iteration, then a new run started before the finalizer throws GeneratorExit), and owns_run = self._active_run is my_active_run at line 1020 keeps the stale finalizer from clearing the successor's run lock. The runner's discard has no such check.
Proposed fix: scope the discard to the run that owns the superstep, for example record a per-run token on the runner when a superstep starts and skip discard() in the finally when the token no longer matches (task cancellation still runs), the same way owns_run guards the cleanup in _run_core. Another option is for a new run to close and await any still-live previous runner generator before it touches State. A regression test modelled on test_workflow_partial_stream_does_not_clobber_successor_active_run that asserts the successor's write survives would pin it. I'm happy to open a PR for this.
Acknowledgements
- I searched existing issues and did not find a duplicate.
- I personally verified this behavior and the reproduction details are authentic.
- I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.
Metadata
Metadata
Labels
Type
Projects
- StatusShow more project fieldsNo status
Activity
[-]Next workflow run loses its state writes when a dropped run is finalized late[/-][+]Python: Next workflow run loses its state writes when a dropped run is finalized late[/+]Python: fix(core): keep a dropped run's late finalizer from discardin…
eavanvalkenburg commentedon Oct 27, 2026
/plan Review the current-main dropped-stream state-ownership bug in coordination with existing PR #9475 (current head e92c3f9), not a parallel implementation. A late abandoned-run finalizer can discard a successor run’s pending writes; maintained partial-stream successor coverage already establishes the supported run-lock boundary. Incorporate Jose Alvarez (@jpalvarezl)’s current review direction and the contributor’s subsequent per-superstep isolation work. Check both late cancellation handlers calling ctx.set_state() and entry-checkpoint snapshots staged before the successor’s first superstep. Define the smallest correct internal ownership boundary across run setup, checkpoint save/restore, commit/discard and failed/cancelled runs, preserving state-copy and superstep semantics plus pending-request/resume behavior. Plan regressions for preserving successor writes, excluding predecessor writes including cancellation cleanup, entry checkpoints and same-run failure cleanup. Do not expand unsupported live-concurrency guarantees. Reuse the ongoing review and contribution; this planning request is not approval of the current patch or an instruction to open a competing PR.
eavanvalkenburg commentedon Nov 11, 2026
/plan Review the current-main dropped-stream state-ownership bug in coordination with existing PR #9475 (current head e92c3f9), not a parallel implementation. A late abandoned-run finalizer can discard a successor run’s pending writes; maintained partial-stream successor coverage already establishes the supported run-lock boundary. Incorporate @jpalvarezl’s current review direction and the contributor’s subsequent per-superstep isolation work. Check both late cancellation handlers calling ctx.set_state() and entry-checkpoint snapshots staged before the successor’s first superstep. Define the smallest correct internal ownership boundary across run setup, checkpoint save/restore, commit/discard and failed/cancelled runs, preserving state-copy and superstep semantics plus pending-request/resume behavior. Plan regressions for preserving successor writes, excluding predecessor writes including cancellation cleanup, entry checkpoints and same-run failure cleanup. Do not expand unsupported live-concurrency guarantees. Reuse the ongoing review and contribution; this planning request is not approval of the current patch or an instruction to open a competing PR.
github-actions commentedon Nov 11, 2026
Summary
A workflow stream that is dropped mid-superstep (a "superstep" is one round of executor work) can be finalized late. If a new run has started on the same
Workflowin the meantime, the old run's cleanup clears the new run's staged state writes. The new run still endsIDLE, so the loss is silent. The report is self-consistent, and I have not run it. This is a Python-only issue.I did not read the open PR #9475 or its reviews, as this workflow requires. The plan below comes from the current
maincode only. Because of that, I cannot say how it lines up with@jpalvarezl's review direction or the contributor's per-superstep work. Please map those onto the steps below.What the code shows
Stateobject is created perWorkflow(_workflow.py~L396). Every run and every executor context shares its single_pendingbuffer (_state.py).finally: if not committed: self._state.discard()(_runner.pyL168-171) clears that shared buffer without checking which run is calling. This is the reported bug._run_corealready guards its own cleanup withowns_runfor the run lock and runtime checkpoint storage (_workflow.py~L1021).test_workflow_partial_stream_does_not_clobber_successor_active_runpins that boundary. State has no equivalent.Two further gaps behind the reported one
finallyand awaits it. If this happens late, an executor'sexcept CancelledErrorhandler can callctx.set_state()after the successor has begun. That write lands in the shared pending buffer and gets committed as the successor's. A token check ondiscard()alone does not stop this. The write has to be unable to reach a buffer the successor owns.create_checkpoint_if_enabled()calls_prepare_checkpoint_state(), which callsself._state.commit()(_runner.py~L248). The entry checkpoint is taken in_execute_with_message_or_checkpointand the response path, before any superstep. A predecessor's leftover pending writes would be committed into the successor's entry checkpoint and committed state. Likewiserestore_from_checkpointandrestore_checkpointcallclear()andimport_state(), which wipe only committed state and the pending buffer as it is at that moment.Recommendation
Code change warranted: this is a silent data loss regression from #8819 (python-1.21.0 did not have it). No workaround helps. The only way to avoid it is to fully drain or close the old stream before starting another run, which the framework does not require.
Keep the scope to this: one run owns its pending writes, and a stale run cannot touch state it does not own. Do not promise anything for truly concurrent runs on one
Workflow. That is still unsupported, and the existing active-run guard still applies.Proposed plan (nothing has been changed yet)
Define ownership. Give each superstep its own pending write buffer, and give the setup/entry-checkpoint/restore phase its own too. Reads see that buffer first, then committed state, as today. Executors would get this scoped view through the existing
send_message(message, state, ctx)path in_run_iteration, soWorkflowContext.set_state/get_stateand the publicStateAPI stay the same. A late handler from an abandoned superstep then writes only to its own orphaned buffer, which nothing commits.Commit and discard on the owner's buffer only. At the superstep boundary, merge that run's buffer into committed state. On failure or cancellation, drop only that buffer. A stale finalizer then has nothing of the successor's to clear. Keep the existing copy-on-set/get and last-write-wins semantics, and keep commit-at-boundary as the visibility rule.
Run setup and checkpoints. Start a fresh ownership scope when a run is set up (message, checkpoint restore, or responses). Make entry-checkpoint capture and
_prepare_checkpoint_statecommit only the new scope's own staged data, never a leftover predecessor buffer. Restore replaces committed state and begins a new scope, so a predecessor can no longer write into it.Pending requests and resume. Executor and edge-runner snapshot keys (
EXECUTOR_STATE_KEY,EDGE_STATE_KEY) are written during checkpoint prep. Keep those writes in the current owner's scope so a resume from a pending request still restores the same data.Regression tests (in
tests/workflow/, next to the partial-stream successor tests):CancelledErrorhandler that callsctx.set_state()after the successor starts does not leak into the successor.Done when the new tests pass, the existing state, checkpoint and HITL (human-in-the-loop) suites are unchanged, and the reproduction prints
b_key = 'from-run-B'.Trade-offs
discard(). It is the smallest approach I found that also closes the cancellation-handler and entry-checkpoint gaps.This is a proposal. I have not run any code or tests.