Repository navigation
Description
Description
A @step that calls another @step works on the first run. After a crash and a resume from a checkpoint, a later call of the inner step returns the cached result of a different call, and its body never runs. The wrong result is returned silently, with no exception. This still reproduces on 1.21.0 when a step argument cannot be fingerprinted (for example a pydantic BaseModel), because MAF falls back to the call-order cache key.
Related: #8292 (closed). The argument fingerprint added in 1.20 fixes the case where all arguments are supported types, but not this one.
Code Sample
import asyncio
import agent_framework as maf
from pydantic import BaseModel
calls: list[str] = []
crash_at = "b"
class Job(BaseModel):
label: str
@maf.step
async def tick(job: Job) -> str:
calls.append(job.label)
if job.label == crash_at:
raise RuntimeError("killed")
return job.label.upper()
@maf.step
async def wrapper(job: Job) -> str:
return await tick(job) # a step called from a step
@maf.workflow(name="repro")
async def repro(_: str) -> list[str]:
return [await wrapper(Job(label="a")), await wrapper(Job(label="b"))]
async def main() -> None:
global crash_at
storage = maf.InMemoryCheckpointStorage()
try:
await repro.build(checkpoint_storage=storage).run("go")
except RuntimeError as exc:
print("first run crashed:", exc)
calls.clear()
crash_at = ""
latest = await storage.get_latest(workflow_name="repro")
result = await repro.build(checkpoint_storage=storage).run(checkpoint_id=latest.checkpoint_id)
print("outputs:", result.get_outputs(), "step bodies run on resume:", calls)
asyncio.run(main())Expected: [['A', 'B']], with tick(b) running on the resume.
Actual (1.18.0 and 1.21.0): [['A', 'A']], and no step body runs on the resume.
With str arguments the same workflow resumes correctly on 1.20.0 and 1.21.0 ([['A', 'B']], only b re-runs), and returns [['A', 'A']] on 1.18.0 and 1.19.0.
Cause (1.18.0)
StepWrapper.__call__ allocates the cache key with ctx._get_step_cache_key(name), which reads and increments _step_call_counters[name], so the key is (name, call order). On a cache hit the step returns without running its body, so steps called inside it never allocate their indices.
- First run:
tick(a)gets("tick", 0),wrapper(a)gets("wrapper", 0),tick(b)gets("tick", 1)and crashes. - Resume:
wrapper(a)is a hit, sotick's counter is not advanced.wrapper(b)runs, and its innertick(b)gets("tick", 0), which is already cached as"A".
Suggestions
- Scope the counters per calling step (for example key
(parent_key, name, idx)), or advance the children's counters on a cache hit. - Or raise a clear error when a step is entered while another step body is running, and document that steps must be flat.
- Or, for arguments that cannot be fingerprinted, fail loudly instead of silently using the order-based key.
The docs do not currently say whether steps may call steps.
Package Versions
agent-framework-core: 1.18.0 and 1.21.0
Python Version
Python 3.12
Metadata
Metadata
Labels
Type
Projects
- StatusShow more project fieldsNo status
Activity
github-actions commentedon Oct 14, 2026
🤖 Automated triage reproduction notes (agent-authored — trust but verify)
Agent analysis
Repro:
python/packages/core/agent_framework/_workflows/_functional.py::StepWrapper.__call__lines 1003-1030 silently reusesRunContext._get_legacy_step_cache_keywhen a checkpointed nested@stepreceives an un-fingerprintable Pydantic model. A cached outer-step hit leaves the inner call-order counter at zero, so the later inner call receives the first call’s cached result. Minimal repro: run nested calls withJob("a")followed by crashingJob("b"), then resume the latest in-memory checkpoint; the output is[['A', 'A']]and the inner body call list is empty.python/packages/core/tests/workflow/test_functional_workflow.py::TestStepReplayIdentity::test_checkpoint_restore_nested_step_with_opaque_arguments1.18.0 and 1.21.01.21.0eavanvalkenburg commentedon Oct 27, 2026
/plan Investigate the current-main silent wrong-result path in nested sequential legacy replay and coordinate with existing PR #9391, not a competing implementation. StepWrapper explicitly documents sequential legacy checkpoint compatibility, and test_legacy_checkpoint_replays_sequential_step pins it. A cached outer step skips the inner call counter; a later inner call can fall back to another invocation’s legacy entry even with fingerprintable arguments or an explicit key when its versioned entry is absent. Separate that supported compatibility/data-integrity failure from the original opaque-argument example, for which replay_key is already documented. Decide a safe fix and any necessary compatibility/migration boundary before accepting #9391’s proposed rejection of all legacy checkpoints; blanket rejection of valid flat sequential legacy replay is not yet agreed. Preserve versioned-hit precedence and cover nested and mixed cached-parent/direct-child cases, automatic and explicit identities, existing flat legacy behavior, new checkpoints, HITL and streaming/non-streaming replay. Do not promise automatic identity for arbitrary opaque inputs.
eavanvalkenburg commentedon Nov 10, 2026
/plan Investigate the current-main silent wrong-result path in nested sequential legacy replay and coordinate with existing PR #9391, not a competing implementation. StepWrapper explicitly documents sequential legacy checkpoint compatibility, and test_legacy_checkpoint_replays_sequential_step pins it. A cached outer step skips the inner call counter; a later inner call can fall back to another invocation’s legacy entry even with fingerprintable arguments or an explicit key when its versioned entry is absent. Separate that supported compatibility/data-integrity failure from the original opaque-argument example, for which replay_key is already documented. Decide a safe fix and any necessary compatibility/migration boundary before accepting #9391’s proposed rejection of all legacy checkpoints; blanket rejection of valid flat sequential legacy replay is not yet agreed. Preserve versioned-hit precedence and cover nested and mixed cached-parent/direct-child cases, automatic and explicit identities, existing flat legacy behavior, new checkpoints, HITL and streaming/non-streaming replay. Do not promise automatic identity for arbitrary opaque inputs.
github-actions commentedon Nov 10, 2026
Summary
The
/planasks me to split this issue in two and decide a safe fix before the open PR (#9391) is accepted. That PR rejects all legacy checkpoints. I did not read #9391's code or reviews, so what follows comes from_functional.pyand the existing tests on main. I have not run anything.Confirmed from the code (
StepWrapper.__call__,RunContext._get_legacy_step_cache_key):name::Nfrom a per-name counter, before anything else.replay_key(explicit) identity, it first looks up its versioned key. If that misses, it still falls back to its legacy key. That fallback is the supported compatibility path for old checkpoints.inner::0. If an old checkpoint holdsinner::0from a different invocation, the call returns that value silently.So there are two separate problems.
A. The original example.
Jobis a Pydantic model with no automatic identity, and there is noreplay_key. The step then uses the legacy key as its only identity, and new checkpoints are written under that key.replay_keyis already the documented answer, in theStepWrapperandstepdocstrings. The remaining gap is that the silent fallback is easy to hit. Failing loudly is reasonable here, but that is a separate, narrower decision from compatibility.B. The compatibility/data-integrity path you describe. The step has a fingerprint or an explicit key. Its versioned entry is absent, as in a checkpoint written by an older version. The call falls back to a legacy entry that belongs to another invocation. The cause is not the identity type. The legacy counters become wrong whenever a cached parent skips its children. Flat sequential replay is pinned by
test_legacy_checkpoint_replays_sequential_step. The existing guards for concurrent legacy hits don't help, because this is sequential. Versioned-hit precedence is already tested (test_versioned_identity_precedes_unrelated_legacy_entry).Recommendation
Code changes are warranted, but narrower than rejecting every legacy checkpoint. The risk is a silent wrong result, so I would not accept a blanket rejection until the alternatives below are ruled out. A blanket rejection would also break resuming valid flat sequential checkpoints from older versions.
I would keep a legacy hit only when the legacy counter is still known to be trustworthy. Once a cached step has been skipped, that is no longer true.
Proposed plan
Track whether legacy counters can still be trusted. Add a "legacy order is still valid" flag to the run context. Because a step may call other steps, I'd use a depth/stack notion of "currently inside a step body", so nested calls are distinguishable from direct ones.
Make the legacy fallback conditional. Keep versioned-key lookup first, unchanged. Then follow these rules on a miss:
ValueErrorthat says to regenerate the checkpoint or addreplay_key. If none exists, run the step live as normal.Write rules for new checkpoints. Steps with an automatic or explicit identity already write versioned keys only, so they are unaffected. Only the opaque-argument case (A) still writes legacy keys. For nested calls with no identity, I'd raise a clear error ("this nested step needs
replay_key") instead of using call order. That error is a behavior change for the flat opaque case too if applied outside nested use, so I would limit it to nested/concurrent situations that already can't be replayed safely. The existing concurrent error is the precedent.Do not promise automatic identity for opaque inputs. Docs should say
replay_keyis required for them. Add a short note that steps called from steps need identity-bearing arguments orreplay_key.Tests, added next to the existing legacy tests:
replay_key, and opaque arguments.request_info) inside nested steps. This matters because cached steps advance the auto-request-info index (_advance_auto_request_info_index_for_cached_step).Docs. Update the
StepWrapper/stepdocstrings and the functional-workflow docs on step-in-step behavior and the legacy boundary.Resolved when: the issue's example either returns
[['A','B']]withtick(b)re-run (withreplay_key) or fails with a clear error (without it). Legacy flat checkpoints still resume. No test finds a silent wrong value.Trade-offs and open decisions:
This is a proposal only, and I have made no changes. Maintainers should confirm the per-parent versus conservative tracking choice and whether the nested opaque case should become an error rather than a legacy-key fallback.