Repository navigation
Description
Observed Behavior
Resuming a workflow from a checkpoint silently discards run kwargs supplied by the caller on that continuation, contrary to the documented contract.
_workflows/_workflow.py:627-636 stages and commits the run kwargs before initial_executor_fn(); _workflows/_runner.py:334-335 then does state.clear(); state.import_state(checkpoint.state), which restores the previous run's kwargs over them. The contract comment at _workflow.py:596-601 says the prior run's kwargs are preserved "unless the caller explicitly provides new kwargs", and cancel_pending_requests shows the intended order by restoring first (_workflow.py:1553-1561).
Real output from the reproduction below:
run 1 outputs: ["message=turn-1 kwargs={'function_invocation_kwargs': {'__global__': {'key': 'OLD'}}}"]
entry checkpoint iteration: 0 state kwargs: {'function_invocation_kwargs': {'__global__': {'key': 'OLD'}}}
run 2 outputs: ["message=turn-1 kwargs={'function_invocation_kwargs': {'__global__': {'key': 'OLD'}}}"]
EXPECTED run 2 kwargs key == NEW (caller-provided kwargs win on continuation)
The discarded kwargs are also re-persisted into later checkpoints, so the continuation permanently runs with the old value.
Expected Behavior
On run(checkpoint_id=..., function_invocation_kwargs={...}) (and the same for client_kwargs, and for checkpoint_id + responses), the caller-supplied kwargs should win over the checkpoint-restored ones, as the contract comment states. Executors reading WORKFLOW_RUN_KWARGS_KEY should see NEW.
Steps to Reproduce
- Build a workflow with checkpoint storage whose executor records
WORKFLOW_RUN_KWARGS_KEYfrom state. - Run 1 with
function_invocation_kwargs={"key": "OLD"}and save the entry checkpoint id. - Rebuild the workflow on a fresh instance and call
run(checkpoint_id=<entry>, function_invocation_kwargs={"key": "NEW"}). - Observe the executor sees
OLD, and later checkpoints keepOLD.
Minimal Reproduction
The full runnable script is repro_kwargs_restore.py (pure Python, mock executor, in-memory checkpoint storage): it prints run 1's kwargs, the entry checkpoint state, and run 2's kwargs with the expected NEW. Happy to attach it to the issue.
Error Messages and Stack Traces
No exception. Actual output:
run 2 outputs: ["message=turn-1 kwargs={'function_invocation_kwargs': {'__global__': {'key': 'OLD'}}}"]
Package Versions
agent-framework-core: 1.23.0 (main at 85f3fc6, 2026-10-29)
Python Version
Python 3.12.2
Operating System
Windows
Regression
Unknown
Additional Context
- Suggested direction: apply caller-supplied kwargs after the checkpoint state is restored (matching
cancel_pending_requests), instead of committing them beforeinitial_executor_fn(). - Searched open and closed issues/PRs: no matching issue or PR. Python: Preserve workflow run kwargs when continuing with
run(responses=...)#4296 fixed responses-only continuation kwargs, Python: [Bug]: Nested workflow cancellation retains stale invocation kwargs #8601/Python: Forward invocation kwargs during nested workflow cancellation #8602 fixed nested-cancellation kwargs; Python: Support workflow entry-state updates and per-AgentExecutor run options #8711/Python: [Feature]: Make workflows stateless with caller-owned checkpoints and msgspec snapshots #8522 are vNext features that do not cover this v1 ordering bug.
AI assistance disclosure: analysis and the reproduction script were prepared with AI assistance (Claude); I personally ran the reproduction and verified the output above.
Acknowledgements: I searched existing issues and did not find a duplicate; I personally verified this behavior; I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.
Metadata
Metadata
Labels
Type
Projects
- StatusShow more project fieldsNo status
Activity
github-actions commentedon Oct 29, 2026
🤖 Automated triage reproduction notes (agent-authored — trust but verify)
Agent analysis
Repro:
python/packages/core/agent_framework/_workflows/_workflow.py::_run_workflow_with_tracingcommits caller kwargs before_runner.py::restore_from_checkpointclears state and imports checkpoint values, triggered byrun(checkpoint_id=..., function_invocation_kwargs=...)orclient_kwargs, with or withoutresponses. Minimal repro: checkpoint a request withkey=OLD, rebuild the workflow, resume withkey=NEW, and assert the resumed executor receivesNEW; it receivesOLD.python/packages/core/tests/workflow/test_workflow_kwargs.py::test_caller_kwargs_override_checkpoint_kwargs_on_restore1.23.01.23.0eavanvalkenburg commentedon Nov 2, 2026
/plan Plan the confirmed checkpoint-resume kwargs precedence bug on current main, coordinating existing #9853 rather than starting a parallel fix. The workflow stages caller overrides before initial checkpoint restoration, then restore clears/imports state and overwrites them, contrary to the documented caller-override contract. Establish restore-before-override ordering for raw/resolved/routed function and client kwargs while preserving checkpoint values when no override is supplied. Cover fresh-instance restore, entry/later checkpoints, checkpoint+responses continuation, omitted/explicit overrides, and parity with cancel_pending_requests. Preserve execution-mode and restored-request validation order, restore failure events/tracing/status, and helper compatibility; do not move restoration outside the protected workflow error/tracing boundary as a shortcut. Use the existing PR fixtures or attach the reporter runnable script to validate the supported matrix. Production scope/ordering implications are uncertain and not minor-ready; keep labels/assignees unchanged and defer new assignment until planning supports a decision.
eavanvalkenburg commentedon Nov 12, 2026
/plan Plan the confirmed checkpoint-resume kwargs precedence bug on current main, coordinating existing #9853 rather than starting a parallel fix. The workflow stages caller overrides before initial checkpoint restoration, then restore clears/imports state and overwrites them, contrary to the documented caller-override contract. Establish restore-before-override ordering for raw/resolved/routed function and client kwargs while preserving checkpoint values when no override is supplied. Cover fresh-instance restore, entry/later checkpoints, checkpoint+responses continuation, omitted/explicit overrides, and parity with cancel_pending_requests. Preserve execution-mode and restored-request validation order, restore failure events/tracing/status, and helper compatibility; do not move restoration outside the protected workflow error/tracing boundary as a shortcut. Use the existing PR fixtures or attach the reporter runnable script to validate the supported matrix. Production scope/ordering implications are uncertain and not minor-ready; keep labels/assignees unchanged and defer new assignment until planning supports a decision.
github-actions commentedon Nov 12, 2026
Summary
When you resume with
run(checkpoint_id=..., function_invocation_kwargs=... / client_kwargs=...), the caller's kwargs are lost. The checkpoint's old kwargs win, and they are saved again into later checkpoints. The same happens withcheckpoint_id+responses. I did not run anything; this is from reading currentmain.Confirmed from the code
_run_workflow_with_tracingin_workflow.py(~L596-640) writes the caller kwargs into state (WORKFLOW_RUN_KWARGS_KEY, the resolved key and the routed key) and commits them. Only after that does it callinitial_executor_fn().initial_executor_fnis_execute_with_message_or_checkpointor_restore_and_send_responses. Both callrunner.restore_from_checkpoint. In_runner.py(~L331) that doesstate.clear()and thenstate.import_state(checkpoint.state). The caller values are wiped and replaced with the old ones._workflow.pysays that on a continuation, prior kwargs are kept "unless the caller explicitly provides new kwargs". The code does not do this.cancel_pending_requests(~L1553-1561) restores the checkpoint first and then runs the tracing path. So the kwargs are written after the restore and the caller wins. That is the behavior the reporter wants._agent_executor.pyand_workflow_executor.py) only read state, so one fix at the write point covers them.Recommendation
Code change is warranted. This is a defect against the documented behavior, not a feature request. A workaround exists but is poor. Callers could use
cancel_pending_requestswithcheckpoint_id, which restores first, but that is not a general replacement for resuming. There is no sample or doc fix that would be enough.I agree with coordinating through #9853 instead of opening a parallel fix. I have not looked at that PR's code, so I am not judging its approach. This plan is based on the repository only.
Proposed plan
This is a proposal. No changes have been made.
Choose where to apply the kwargs. They must be written after the restore and before the first executor runs. The restore is currently inside
initial_executor_fn, which is inside the protected error/tracing block. That block is what emits the failed event, theFAILEDstatus and the trace error. So do not move the restore outside it. Two options:_run_workflow_with_tracing, either beforeinitial_executor_fn(when there is no restore) or after the restore step inside it.restore_from_checkpoint.Either way, keep the existing order of checks: mode checks, the pending-request warning and
_validate_responses_internalstay first. The restored-request validation in_send_responses_internalhappens after restore, so applying kwargs between restore and sending responses must not change what errors appear.Keep the three state keys and the raw keys consistent. Raw function kwargs, raw client kwargs, resolved kwargs and routed kwargs are all written together today. Keep that, so no executor can see a half-new, half-old mix.
Preserve the no-override case. If the caller passes no kwargs, the restored checkpoint values must stay as they are. The fresh-message path (
not is_continuation) still resets to{}.Keep
cancel_pending_requestsbehavior unchanged, and use it as the reference. It callsrestore_from_checkpointitself and then enters the tracing path withinitial_executor_fn=apply_cancellations, so the new ordering must not apply kwargs twice or skip them there.Leave the nested restore alone.
WorkflowExecutorcallsrunner.restore_checkpoint(...)for sub-workflows. It uses its own propagation of the parent's kwargs, so I would not change it unless tests show it is affected.Tests in
python/packages/core/tests/workflow/test_workflow_kwargs.py(the file already hastest_caller_kwargs_override_checkpoint_kwargs_on_restorefrom triage and file-checkpoint tests). Cover:checkpoint_id+responses;cancel_pending_requests;The reporter's script can be turned into a test or used for a manual check if the maintainers want it attached.
Resolved after review — success looks like the reporter's repro printing
NEWon run 2 and in later checkpoints, with all existing workflow tests passing.Questions to settle in the PR (not blockers for the plan)
function_invocation_kwargsreplaces the whole stored dict, so previously storedclient_kwargsare dropped. After restoring, should an override replace everything (current behavior on a fresh run) or merge per kind? I would keep the replace-all behavior to match the present semantics, but a maintainer should confirm, since restore now brings old values into play.The assignee and labels are unchanged. I'd leave the assignment decision until the ordering approach above is confirmed.