Skip to content

Python: [Bug]: Resuming from a checkpoint discards caller-supplied run kwargs, contrary to the documented contract #9850

Description

@BlueX888

Observed Behavior

Resuming a workflow from a checkpoint silently discards run kwargs supplied by the caller on that continuation, contrary to the documented contract.

_workflows/_workflow.py:627-636 stages and commits the run kwargs before initial_executor_fn(); _workflows/_runner.py:334-335 then does state.clear(); state.import_state(checkpoint.state), which restores the previous run's kwargs over them. The contract comment at _workflow.py:596-601 says the prior run's kwargs are preserved "unless the caller explicitly provides new kwargs", and cancel_pending_requests shows the intended order by restoring first (_workflow.py:1553-1561).

Real output from the reproduction below:

run 1 outputs: ["message=turn-1 kwargs={'function_invocation_kwargs': {'__global__': {'key': 'OLD'}}}"]
entry checkpoint iteration: 0 state kwargs: {'function_invocation_kwargs': {'__global__': {'key': 'OLD'}}}
run 2 outputs: ["message=turn-1 kwargs={'function_invocation_kwargs': {'__global__': {'key': 'OLD'}}}"]
EXPECTED run 2 kwargs key == NEW (caller-provided kwargs win on continuation)

The discarded kwargs are also re-persisted into later checkpoints, so the continuation permanently runs with the old value.

Expected Behavior

On run(checkpoint_id=..., function_invocation_kwargs={...}) (and the same for client_kwargs, and for checkpoint_id + responses), the caller-supplied kwargs should win over the checkpoint-restored ones, as the contract comment states. Executors reading WORKFLOW_RUN_KWARGS_KEY should see NEW.

Steps to Reproduce

  1. Build a workflow with checkpoint storage whose executor records WORKFLOW_RUN_KWARGS_KEY from state.
  2. Run 1 with function_invocation_kwargs={"key": "OLD"} and save the entry checkpoint id.
  3. Rebuild the workflow on a fresh instance and call run(checkpoint_id=<entry>, function_invocation_kwargs={"key": "NEW"}).
  4. Observe the executor sees OLD, and later checkpoints keep OLD.

Minimal Reproduction

The full runnable script is repro_kwargs_restore.py (pure Python, mock executor, in-memory checkpoint storage): it prints run 1's kwargs, the entry checkpoint state, and run 2's kwargs with the expected NEW. Happy to attach it to the issue.

Error Messages and Stack Traces

No exception. Actual output:

run 2 outputs: ["message=turn-1 kwargs={'function_invocation_kwargs': {'__global__': {'key': 'OLD'}}}"]

Package Versions

agent-framework-core: 1.23.0 (main at 85f3fc6, 2026-10-29)

Python Version

Python 3.12.2

Operating System

Windows

Regression

Unknown

Additional Context

AI assistance disclosure: analysis and the reproduction script were prepared with AI assistance (Claude); I personally ran the reproduction and verified the output above.

Acknowledgements: I searched existing issues and did not find a duplicate; I personally verified this behavior; I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.

Activity

  1. added
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Oct 29, 2026
  2. added
    reproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflow
    on Oct 29, 2026
  3. github-actions commented on Oct 29, 2026

    @github-actions
    Contributor

    🤖 Automated triage reproduction notes (agent-authored — trust but verify)

    Agent analysis

    Repro: python/packages/core/agent_framework/_workflows/_workflow.py::_run_workflow_with_tracing commits caller kwargs before _runner.py::restore_from_checkpoint clears state and imports checkpoint values, triggered by run(checkpoint_id=..., function_invocation_kwargs=...) or client_kwargs, with or without responses. Minimal repro: checkpoint a request with key=OLD, rebuild the workflow, resume with key=NEW, and assert the resumed executor receives NEW; it receives OLD.

    • Failing test: python/packages/core/tests/workflow/test_workflow_kwargs.py::test_caller_kwargs_override_checkpoint_kwargs_on_restore
    • Files examined: python/packages/core/agent_framework/_workflows/_workflow.py, python/packages/core/agent_framework/_workflows/_runner.py, python/packages/core/tests/workflow/test_workflow_kwargs.py, python/packages/core/pyproject.toml
    • Tests run: test_caller_kwargs_override_checkpoint_kwargs_on_restore, test_collision_free_kwargs_survive_file_checkpoint_restore
    • Reported version: 1.23.0
    • Current version: 1.23.0
  4. added
    workflowsUsage: [Issues, PRs], Target: Workflows
    and removed
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Oct 29, 2026
  5. eavanvalkenburg commented on Nov 2, 2026

    @eavanvalkenburg
    Member

    /plan Plan the confirmed checkpoint-resume kwargs precedence bug on current main, coordinating existing #9853 rather than starting a parallel fix. The workflow stages caller overrides before initial checkpoint restoration, then restore clears/imports state and overwrites them, contrary to the documented caller-override contract. Establish restore-before-override ordering for raw/resolved/routed function and client kwargs while preserving checkpoint values when no override is supplied. Cover fresh-instance restore, entry/later checkpoints, checkpoint+responses continuation, omitted/explicit overrides, and parity with cancel_pending_requests. Preserve execution-mode and restored-request validation order, restore failure events/tracing/status, and helper compatibility; do not move restoration outside the protected workflow error/tracing boundary as a shortcut. Use the existing PR fixtures or attach the reporter runnable script to validate the supported matrix. Production scope/ordering implications are uncertain and not minor-ready; keep labels/assignees unchanged and defer new assignment until planning supports a decision.

  6. eavanvalkenburg commented on Nov 12, 2026

    @eavanvalkenburg
    Member

    /plan Plan the confirmed checkpoint-resume kwargs precedence bug on current main, coordinating existing #9853 rather than starting a parallel fix. The workflow stages caller overrides before initial checkpoint restoration, then restore clears/imports state and overwrites them, contrary to the documented caller-override contract. Establish restore-before-override ordering for raw/resolved/routed function and client kwargs while preserving checkpoint values when no override is supplied. Cover fresh-instance restore, entry/later checkpoints, checkpoint+responses continuation, omitted/explicit overrides, and parity with cancel_pending_requests. Preserve execution-mode and restored-request validation order, restore failure events/tracing/status, and helper compatibility; do not move restoration outside the protected workflow error/tracing boundary as a shortcut. Use the existing PR fixtures or attach the reporter runnable script to validate the supported matrix. Production scope/ordering implications are uncertain and not minor-ready; keep labels/assignees unchanged and defer new assignment until planning supports a decision.

  7. github-actions commented on Nov 12, 2026

    @github-actions
    Contributor

    Summary

    When you resume with run(checkpoint_id=..., function_invocation_kwargs=... / client_kwargs=...), the caller's kwargs are lost. The checkpoint's old kwargs win, and they are saved again into later checkpoints. The same happens with checkpoint_id + responses. I did not run anything; this is from reading current main.

    Confirmed from the code

    • _run_workflow_with_tracing in _workflow.py (~L596-640) writes the caller kwargs into state (WORKFLOW_RUN_KWARGS_KEY, the resolved key and the routed key) and commits them. Only after that does it call initial_executor_fn().
    • For a checkpoint resume, initial_executor_fn is _execute_with_message_or_checkpoint or _restore_and_send_responses. Both call runner.restore_from_checkpoint. In _runner.py (~L331) that does state.clear() and then state.import_state(checkpoint.state). The caller values are wiped and replaced with the old ones.
    • The comment in _workflow.py says that on a continuation, prior kwargs are kept "unless the caller explicitly provides new kwargs". The code does not do this.
    • cancel_pending_requests (~L1553-1561) restores the checkpoint first and then runs the tracing path. So the kwargs are written after the restore and the caller wins. That is the behavior the reporter wants.
    • Other readers of the kwargs (_agent_executor.py and _workflow_executor.py) only read state, so one fix at the write point covers them.

    Recommendation

    Code change is warranted. This is a defect against the documented behavior, not a feature request. A workaround exists but is poor. Callers could use cancel_pending_requests with checkpoint_id, which restores first, but that is not a general replacement for resuming. There is no sample or doc fix that would be enough.

    I agree with coordinating through #9853 instead of opening a parallel fix. I have not looked at that PR's code, so I am not judging its approach. This plan is based on the repository only.

    Proposed plan

    This is a proposal. No changes have been made.

    1. Choose where to apply the kwargs. They must be written after the restore and before the first executor runs. The restore is currently inside initial_executor_fn, which is inside the protected error/tracing block. That block is what emits the failed event, the FAILED status and the trace error. So do not move the restore outside it. Two options:

      • Split the "stage kwargs" step into a helper. Call it from _run_workflow_with_tracing, either before initial_executor_fn (when there is no restore) or after the restore step inside it.
      • Or pass the kwargs in as a callback. The restore paths call it right after restore_from_checkpoint.

      Either way, keep the existing order of checks: mode checks, the pending-request warning and _validate_responses_internal stay first. The restored-request validation in _send_responses_internal happens after restore, so applying kwargs between restore and sending responses must not change what errors appear.

    2. Keep the three state keys and the raw keys consistent. Raw function kwargs, raw client kwargs, resolved kwargs and routed kwargs are all written together today. Keep that, so no executor can see a half-new, half-old mix.

    3. Preserve the no-override case. If the caller passes no kwargs, the restored checkpoint values must stay as they are. The fresh-message path (not is_continuation) still resets to {}.

    4. Keep cancel_pending_requests behavior unchanged, and use it as the reference. It calls restore_from_checkpoint itself and then enters the tracing path with initial_executor_fn=apply_cancellations, so the new ordering must not apply kwargs twice or skip them there.

    5. Leave the nested restore alone. WorkflowExecutor calls runner.restore_checkpoint(...) for sub-workflows. It uses its own propagation of the parent's kwargs, so I would not change it unless tests show it is affected.

    6. Tests in python/packages/core/tests/workflow/test_workflow_kwargs.py (the file already has test_caller_kwargs_override_checkpoint_kwargs_on_restore from triage and file-checkpoint tests). Cover:

      • a fresh instance restoring from the entry checkpoint and from a later checkpoint;
      • checkpoint_id + responses;
      • override omitted (old value kept) and override supplied (new value seen by executors and saved in later checkpoints);
      • function kwargs only, client kwargs only, both, plain mappings, and the routed/executor-specific form;
      • parity with cancel_pending_requests;
      • restore failure (missing or mismatched checkpoint) still emits failed event/status and trace error, and does not leave caller kwargs half-applied;
      • invalid mode combinations and unknown response IDs still fail with the same errors.

      The reporter's script can be turned into a test or used for a manual check if the maintainers want it attached.

    7. Resolved after review — success looks like the reporter's repro printing NEW on run 2 and in later checkpoints, with all existing workflow tests passing.

    Questions to settle in the PR (not blockers for the plan)

    • Partial overrides. Today, passing only function_invocation_kwargs replaces the whole stored dict, so previously stored client_kwargs are dropped. After restoring, should an override replace everything (current behavior on a fresh run) or merge per kind? I would keep the replace-all behavior to match the present semantics, but a maintainer should confirm, since restore now brings old values into play.
    • Compatibility. Anyone who relied on checkpoint values winning would see a change. This contradicts the documented contract, so I'd treat it as a fix, but it is worth a note in the PR description.

    The assignee and labels are unchanged. I'd leave the assignment decision until the ordering approach above is confirmed.

    Generated by Issue Planning for #9850 · copilot · auto · ~42 AIC · ⌖ ~1 AIC · ⊞ 4.8K · ◷

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

pythonUsage: [Issues, PRs], Target: Pythonready-for-implementationMaintainers agree the problem and direction are ready for implementation.reproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflowworkflowsUsage: [Issues, PRs], Target: Workflows

Type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions