Repository navigation
Python: Keep a dropped run's finalizer from discarding state writes #9475
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
4564623
cc54a14
f10c1bf
e92c3f9
9af013f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -589,6 +589,11 @@ async def _run_workflow_with_tracing( | |
| if not is_continuation: | ||
| self._runner.reset_iteration_count() | ||
|
|
||
| # Pending writes here come from a superstep or checkpoint save that never committed; do not | ||
| # commit them, and ignore later writes from that superstep's executors if it is still finalizing. | ||
| self._runner.state.begin_epoch() | ||
| self._runner.state.discard() | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This discard only removes predecessor writes already pending when B reaches this point. A's finalizer can run later: its cancelled handler may call
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks, good catch. Fixed in e92c3f9: pending writes are now scoped per superstep, so the dropped run's cleanup write is ignored. New test:
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks, good catch. Fixed in e92c3f9: pending writes are now scoped per superstep, so the dropped run's cleanup write is ignored. Regression: test_workflow_partial_stream_cancellation_write_not_committed_by_successor. |
||
|
|
||
| # Store run kwargs in State so executors can access them. | ||
| # Per-run kwargs semantics: | ||
| # - On a fresh message run, prior kwargs go away (set to {} | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
digit50 Stale contexts can still mutate successor state through other State methods — WorkflowContext.state exposes the shared State object, but epoch checks are only applied to set, delete, and clear. A late cancellation handler from a dropped run can call ctx.state.discard() after the successor has staged writes, clearing those writes; commit() can publish them before the superstep boundary, and import_state() can directly alter committed state. These mutations bypass the ownership guard and let a stale run affect its successor.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now fixed in 9af013f:
commit,discardandimport_stateget the same stale-epoch check asset,deleteandclear, and so doesbegin_epoch, which a stale handler could otherwise call to make the successor's own writes look stale. Framework callers of these methods all run outside any superstep's bound context, so they are unaffected. Regression:test_workflow_partial_stream_cancellation_state_operation_ignored_for_successor, parametrized over the four methods.