Skip to content

Python: [Bug]: Functional workflow checkpoint replay returns a wrong cached result when a @step calls another @step #9385

Open

Description

Description

A @step that calls another @step works on the first run. After a crash and a resume from a checkpoint, a later call of the inner step returns the cached result of a different call, and its body never runs. The wrong result is returned silently, with no exception. This still reproduces on 1.21.0 when a step argument cannot be fingerprinted (for example a pydantic BaseModel), because MAF falls back to the call-order cache key.

Related: #8292 (closed). The argument fingerprint added in 1.20 fixes the case where all arguments are supported types, but not this one.

Code Sample

import asyncio

import agent_framework as maf
from pydantic import BaseModel

calls: list[str] = []
crash_at = "b"


class Job(BaseModel):
    label: str


@maf.step
async def tick(job: Job) -> str:
    calls.append(job.label)
    if job.label == crash_at:
        raise RuntimeError("killed")
    return job.label.upper()


@maf.step
async def wrapper(job: Job) -> str:
    return await tick(job)  # a step called from a step


@maf.workflow(name="repro")
async def repro(_: str) -> list[str]:
    return [await wrapper(Job(label="a")), await wrapper(Job(label="b"))]


async def main() -> None:
    global crash_at
    storage = maf.InMemoryCheckpointStorage()
    try:
        await repro.build(checkpoint_storage=storage).run("go")
    except RuntimeError as exc:
        print("first run crashed:", exc)

    calls.clear()
    crash_at = ""
    latest = await storage.get_latest(workflow_name="repro")
    result = await repro.build(checkpoint_storage=storage).run(checkpoint_id=latest.checkpoint_id)
    print("outputs:", result.get_outputs(), "step bodies run on resume:", calls)


asyncio.run(main())

Expected: [['A', 'B']], with tick(b) running on the resume.
Actual (1.18.0 and 1.21.0): [['A', 'A']], and no step body runs on the resume.

With str arguments the same workflow resumes correctly on 1.20.0 and 1.21.0 ([['A', 'B']], only b re-runs), and returns [['A', 'A']] on 1.18.0 and 1.19.0.

Cause (1.18.0)

StepWrapper.__call__ allocates the cache key with ctx._get_step_cache_key(name), which reads and increments _step_call_counters[name], so the key is (name, call order). On a cache hit the step returns without running its body, so steps called inside it never allocate their indices.

  • First run: tick(a) gets ("tick", 0), wrapper(a) gets ("wrapper", 0), tick(b) gets ("tick", 1) and crashes.
  • Resume: wrapper(a) is a hit, so tick's counter is not advanced. wrapper(b) runs, and its inner tick(b) gets ("tick", 0), which is already cached as "A".

Suggestions

  1. Scope the counters per calling step (for example key (parent_key, name, idx)), or advance the children's counters on a cache hit.
  2. Or raise a clear error when a step is entered while another step body is running, and document that steps must be flat.
  3. Or, for arguments that cannot be fingerprinted, fail loudly instead of silently using the order-based key.

The docs do not currently say whether steps may call steps.

Package Versions

agent-framework-core: 1.18.0 and 1.21.0

Python Version

Python 3.12

Activity

  1. added
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Oct 14, 2026
  2. added
    reproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflow
    on Oct 14, 2026
  3. github-actions commented on Oct 14, 2026

    @github-actions
    Contributor

    🤖 Automated triage reproduction notes (agent-authored — trust but verify)

    Agent analysis

    Repro: python/packages/core/agent_framework/_workflows/_functional.py::StepWrapper.__call__ lines 1003-1030 silently reuses RunContext._get_legacy_step_cache_key when a checkpointed nested @step receives an un-fingerprintable Pydantic model. A cached outer-step hit leaves the inner call-order counter at zero, so the later inner call receives the first call’s cached result. Minimal repro: run nested calls with Job("a") followed by crashing Job("b"), then resume the latest in-memory checkpoint; the output is [['A', 'A']] and the inner body call list is empty.

    • Failing test: python/packages/core/tests/workflow/test_functional_workflow.py::TestStepReplayIdentity::test_checkpoint_restore_nested_step_with_opaque_arguments
    • Files examined: python/packages/core/agent_framework/_workflows/_functional.py, python/packages/core/tests/workflow/test_functional_workflow.py, python/packages/core/pyproject.toml
    • Tests run: test_checkpoint_restore_nested_step_with_opaque_arguments (failed), test_per_step_checkpoint_enables_crash_recovery (passed), test_concurrent_opaque_arguments_require_replay_key (passed), test_checkpoint_restore_preserves_concurrent_step_invocations (passed)
    • Reported version: 1.18.0 and 1.21.0
    • Current version: 1.21.0
  4. added
    workflowsUsage: [Issues, PRs], Target: Workflows
    and removed
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Oct 14, 2026
  5. eavanvalkenburg commented on Oct 27, 2026

    @eavanvalkenburg
    Member

    /plan Investigate the current-main silent wrong-result path in nested sequential legacy replay and coordinate with existing PR #9391, not a competing implementation. StepWrapper explicitly documents sequential legacy checkpoint compatibility, and test_legacy_checkpoint_replays_sequential_step pins it. A cached outer step skips the inner call counter; a later inner call can fall back to another invocation’s legacy entry even with fingerprintable arguments or an explicit key when its versioned entry is absent. Separate that supported compatibility/data-integrity failure from the original opaque-argument example, for which replay_key is already documented. Decide a safe fix and any necessary compatibility/migration boundary before accepting #9391’s proposed rejection of all legacy checkpoints; blanket rejection of valid flat sequential legacy replay is not yet agreed. Preserve versioned-hit precedence and cover nested and mixed cached-parent/direct-child cases, automatic and explicit identities, existing flat legacy behavior, new checkpoints, HITL and streaming/non-streaming replay. Do not promise automatic identity for arbitrary opaque inputs.

  6. eavanvalkenburg commented on Nov 10, 2026

    @eavanvalkenburg
    Member

    /plan Investigate the current-main silent wrong-result path in nested sequential legacy replay and coordinate with existing PR #9391, not a competing implementation. StepWrapper explicitly documents sequential legacy checkpoint compatibility, and test_legacy_checkpoint_replays_sequential_step pins it. A cached outer step skips the inner call counter; a later inner call can fall back to another invocation’s legacy entry even with fingerprintable arguments or an explicit key when its versioned entry is absent. Separate that supported compatibility/data-integrity failure from the original opaque-argument example, for which replay_key is already documented. Decide a safe fix and any necessary compatibility/migration boundary before accepting #9391’s proposed rejection of all legacy checkpoints; blanket rejection of valid flat sequential legacy replay is not yet agreed. Preserve versioned-hit precedence and cover nested and mixed cached-parent/direct-child cases, automatic and explicit identities, existing flat legacy behavior, new checkpoints, HITL and streaming/non-streaming replay. Do not promise automatic identity for arbitrary opaque inputs.

  7. github-actions commented on Nov 10, 2026

    @github-actions
    Contributor

    Summary

    The /plan asks me to split this issue in two and decide a safe fix before the open PR (#9391) is accepted. That PR rejects all legacy checkpoints. I did not read #9391's code or reviews, so what follows comes from _functional.py and the existing tests on main. I have not run anything.

    Confirmed from the code (StepWrapper.__call__, RunContext._get_legacy_step_cache_key):

    • Every step call allocates a legacy key name::N from a per-name counter, before anything else.
    • If the step has a fingerprint (automatic) or replay_key (explicit) identity, it first looks up its versioned key. If that misses, it still falls back to its legacy key. That fallback is the supported compatibility path for old checkpoints.
    • When an outer step is a cache hit, its body never runs. The inner steps' counters are therefore not advanced.
    • On resume, a later inner call gets inner::0. If an old checkpoint holds inner::0 from a different invocation, the call returns that value silently.

    So there are two separate problems.

    A. The original example. Job is a Pydantic model with no automatic identity, and there is no replay_key. The step then uses the legacy key as its only identity, and new checkpoints are written under that key. replay_key is already the documented answer, in the StepWrapper and step docstrings. The remaining gap is that the silent fallback is easy to hit. Failing loudly is reasonable here, but that is a separate, narrower decision from compatibility.

    B. The compatibility/data-integrity path you describe. The step has a fingerprint or an explicit key. Its versioned entry is absent, as in a checkpoint written by an older version. The call falls back to a legacy entry that belongs to another invocation. The cause is not the identity type. The legacy counters become wrong whenever a cached parent skips its children. Flat sequential replay is pinned by test_legacy_checkpoint_replays_sequential_step. The existing guards for concurrent legacy hits don't help, because this is sequential. Versioned-hit precedence is already tested (test_versioned_identity_precedes_unrelated_legacy_entry).

    Recommendation

    Code changes are warranted, but narrower than rejecting every legacy checkpoint. The risk is a silent wrong result, so I would not accept a blanket rejection until the alternatives below are ruled out. A blanket rejection would also break resuming valid flat sequential checkpoints from older versions.

    I would keep a legacy hit only when the legacy counter is still known to be trustworthy. Once a cached step has been skipped, that is no longer true.

    Proposed plan

    1. Track whether legacy counters can still be trusted. Add a "legacy order is still valid" flag to the run context. Because a step may call other steps, I'd use a depth/stack notion of "currently inside a step body", so nested calls are distinguishable from direct ones.

    2. Make the legacy fallback conditional. Keep versioned-key lookup first, unchanged. Then follow these rules on a miss:

      • Flat call, nothing skipped yet: allow the legacy hit, as today. Existing flat behavior and its tests are preserved.
      • Nested call (made while another step body is running) or a call after any cached step with children was skipped: the order can no longer be trusted. Do not use the legacy entry. If one exists, raise a clear ValueError that says to regenerate the checkpoint or add replay_key. If none exists, run the step live as normal.
      • Open design choice: whether "after a skipped parent" should be tracked per parent or globally. Per-parent is more permissive and more complex, because a cached parent has no record of whether it had children. The conservative option is to treat any legacy hit that follows a cached step as unsafe if a step body has ever called steps in this run. The maintainers should pick one.
    3. Write rules for new checkpoints. Steps with an automatic or explicit identity already write versioned keys only, so they are unaffected. Only the opaque-argument case (A) still writes legacy keys. For nested calls with no identity, I'd raise a clear error ("this nested step needs replay_key") instead of using call order. That error is a behavior change for the flat opaque case too if applied outside nested use, so I would limit it to nested/concurrent situations that already can't be replayed safely. The existing concurrent error is the precedent.

    4. Do not promise automatic identity for opaque inputs. Docs should say replay_key is required for them. Add a short note that steps called from steps need identity-bearing arguments or replay_key.

    5. Tests, added next to the existing legacy tests:

      • Nested, cached parent, then a direct child call that would hit another invocation's legacy entry. Cover automatic identity, explicit replay_key, and opaque arguments.
      • Mixed cached-parent plus direct-child sequences.
      • Flat legacy replay is unchanged. A versioned entry still wins over a conflicting legacy one.
      • A new checkpoint written by nested steps resumes correctly.
      • HITL (request_info) inside nested steps. This matters because cached steps advance the auto-request-info index (_advance_auto_request_info_index_for_cached_step).
      • Streaming and non-streaming resume paths.
    6. Docs. Update the StepWrapper/step docstrings and the functional-workflow docs on step-in-step behavior and the legacy boundary.

    Resolved when: the issue's example either returns [['A','B']] with tick(b) re-run (with replay_key) or fails with a clear error (without it). Legacy flat checkpoints still resume. No test finds a silent wrong value.

    Trade-offs and open decisions:

    • Blanket rejection ([BREAKING] Python: Reject unsafe functional step replay #9391's direction): simpler and stricter, but breaks flat legacy resume. That is a documented compatibility promise, so it needs an explicit maintainer decision and a migration note.
    • Scoped rejection (proposed above): keeps compatibility but needs the extra "is order still trusted" state.
    • Migration boundary: I'd treat legacy support as "flat sequential only", say so in the docs, and consider a later deprecation of legacy entries once they are old enough.

    This is a proposal only, and I have made no changes. Maintainers should confirm the per-parent versus conservative tracking choice and whether the nested opaque case should become an error rather than a legacy-key fallback.

    Generated by Issue Planning for #9385 · copilot · auto · ~39 AIC · ⌖ ~1 AIC · ⊞ 5K · ◷

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

pythonUsage: [Issues, PRs], Target: PythonreproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflowworkflowsUsage: [Issues, PRs], Target: Workflows

Type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions