Skip to content

Python: [Bug]: Declarative workflows cannot save or resume checkpoints with FileCheckpointStorage #10074

Description

Observed Behavior

A declarative workflow that uses FileCheckpointStorage (or any storage that decodes through the restricted checkpoint unpickler, such as CosmosCheckpointStorage) saves only its first checkpoint. Every later checkpoint fails with:

Failed to create checkpoint at iteration 1: Failed to decode pickled checkpoint data: Checkpoint deserialization blocked for type 'agent_framework_declarative._workflows._declarative_base:ActionComplete'. ...
Note that this does not fail the workflow run.

The run itself continues, so nothing raises. But the checkpoint taken while the workflow waits for input is never stored, so a paused workflow can't be resumed from storage. Restoring the latest checkpoint returns the iteration-0 one, and sending the response then fails with ValueError: Response provided for unknown request ID.

The cause is the same one #8258 fixed for orchestrations. The restricted unpickler trusts the agent_framework. and agent_framework_orchestrations. prefixes, but not agent_framework_declarative. Every declarative executor sends ActionComplete / ActionTrigger envelopes, and pauses checkpoint ExternalInputRequest, ToolApprovalRequest and the other request types from this package.

The only workaround today is to pass framework-internal module paths in allowed_checkpoint_types. test_workflow_run_kwargs_survive_file_checkpoint_restore in packages/declarative/tests/test_graph_executors.py does exactly that. samples/03-workflows/declarative/function_tools/main.py uses FileCheckpointStorage(TEMP_DIR) without that workaround and hits this. The WorkflowFactory docstring's checkpointing example ("With checkpointing for pause/resume support") does too, and it also passes path=, which FileCheckpointStorage doesn't accept.

Expected Behavior

Declarative workflows checkpoint and resume with FileCheckpointStorage without a user-maintained allowlist, the same way orchestrations do after #8258.

Steps to Reproduce

  1. Build a declarative workflow with a RequestExternalInput action and WorkflowFactory(checkpoint_storage=FileCheckpointStorage(dir)).
  2. Run it until it pauses.
  3. Count the checkpoint files and restore the latest one into a fresh workflow.
  4. Send the response for the pending request.

Minimal Reproduction

import asyncio, os, tempfile
from agent_framework import FileCheckpointStorage
from agent_framework.declarative import ExternalInputResponse, WorkflowFactory

YAML = """
name: approval-flow
actions:
  - kind: SetValue
    path: Local.greeting
    value: Hello
  - kind: RequestExternalInput
    id: ask
    prompt: Approve?
    variable: Local.answer
  - kind: SendActivity
    activity:
      text: '=Concatenate(Local.greeting, " - answer ", Local.answer)'
"""

async def main():
    with tempfile.TemporaryDirectory() as d:
        storage = FileCheckpointStorage(d)
        wf = WorkflowFactory(checkpoint_storage=storage).create_workflow_from_yaml(YAML)
        request = (await wf.run({})).get_request_info_events()[0]
        print("checkpoint files:", len([f for f in os.listdir(d) if f.endswith(".json")]))
        latest = await storage.get_latest(workflow_name=wf.name)
        restored = WorkflowFactory(checkpoint_storage=storage).create_workflow_from_yaml(YAML)
        await restored.run(checkpoint_id=latest.checkpoint_id)
        result = await restored.run(responses={request.request_id: ExternalInputResponse(user_input="yes")})
        print(result.get_outputs())

asyncio.run(main())

Error Messages and Stack Traces

WARNING Failed to create checkpoint at iteration 1: Failed to decode pickled checkpoint data: Checkpoint deserialization blocked for type 'agent_framework_declarative._workflows._declarative_base:ActionComplete'. ...
WARNING Failed to create checkpoint at iteration 2: ...
WARNING Failed to create checkpoint at iteration 3: ...
checkpoint files: 1
ValueError: Response provided for unknown request ID: cc60e75d-...

With the declarative types allowed, the same script writes 4 checkpoints and prints ['Hello - answer yes'].

Package Versions

agent-framework-declarative: 1.4.1; main at dc6617a

Python Version

Python 3.13

Operating System

macOS

Regression

Unknown

Additional Context

Proposed fix, mirroring #8258: register the framework-owned declarative types that cross a checkpoint boundary through the public register_checkpoint_type when agent_framework_declarative is imported. These are the executor message envelopes (ActionTrigger, ActionComplete, ConditionResult, LoopIterationResult, LoopControl); the request_info payloads and responses (ExternalInputRequest/Response, AgentExternalInputRequest/Response, ToolApprovalRequest/Response, MCPToolApprovalRequest); and ExternalLoopState, which is kept in workflow state while an agent's external input loop is paused. This keeps the core prefix allowlist unchanged. I have a PR ready and will link it here.

Activity

  1. added
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Nov 4, 2026
  2. 1aifanatic commented on Nov 4, 2026

    @1aifanatic
    ContributorAuthor

    PR with the proposed fix and tests: #10076

  3. added
    reproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflow
    on Nov 4, 2026
  4. github-actions commented on Nov 4, 2026

    @github-actions
    Contributor

    🤖 Automated triage reproduction notes (agent-authored — trust but verify)

    Agent analysis

    Repro: declarative checkpoint integration in python/packages/declarative/agent_framework_declarative/_workflows/__init__.py does not register its framework-owned payload types, so _RestrictedUnpickler.find_class in python/packages/core/agent_framework/_workflows/_checkpoint_encoding.py:328 rejects ActionComplete and ExternalInputRequest. Triggering a declarative RequestExternalInput workflow with FileCheckpointStorage leaves only iteration 0, and restoring it causes the original response ID to be rejected. Minimal repro: create a checkpointed YAML workflow containing SetValue followed by RequestExternalInput, run until paused, restore get_latest() into a fresh workflow, and submit the original ExternalInputResponse.

    • Failing test: python/packages/declarative/tests/test_issue_10074.py
    • Files examined: python/packages/core/agent_framework/_workflows/_checkpoint_encoding.py, python/packages/core/agent_framework/_workflows/_checkpoint.py, python/packages/declarative/agent_framework_declarative/init.py, python/packages/declarative/agent_framework_declarative/_workflows/_declarative_base.py, python/packages/declarative/agent_framework_declarative/_workflows/_factory.py, python/packages/declarative/tests/test_graph_executors.py, python/packages/declarative/pyproject.toml
    • Tests run: test_graph_executors.py -k workflow_run_kwargs_survive_file_checkpoint_restore (4 passed), test_issue_10074.py (2 failed)
    • Reported version: 1.4.1
    • Current version: 1.4.1
  5. added
    declarativeUsage: [Issues, PRs], Target: declarative agents and workflows
    and removed
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Nov 4, 2026
  6. eavanvalkenburg commented on Nov 9, 2026

    @eavanvalkenburg
    Member

    /plan Investigate the declarative checkpoint type-registration gap using the existing register_checkpoint_type mechanism and the orchestration precedent in #8258. Current main independently reproduced only one saved checkpoint, blocked ActionComplete/ExternalInputRequest, and an unknown-request-ID failure after restoring the stale checkpoint; an explicit scoped allowlist produced four checkpoints and a successful fresh-workflow resume. Establish the complete minimal set of framework-owned message, request/response, nested, and state types crossing checkpoint boundaries. Preserve the restricted unpickling contract and rejection of unrelated custom types; do not broaden package-name prefixes. Assess process-wide import-time registration and File/Cosmos shared-codec behavior, with focused round-trip and fresh-workflow pause/resume tests including input and approval paths. Review PR #10076 as an existing proposal, not implementation approval. The production change is near the minor-size threshold, but trust-surface completeness and durable-resume correctness must be settled before readiness.

  7. github-actions commented on Nov 9, 2026

    @github-actions
    Contributor

    Summary

    Declarative workflows (Python) only store their first checkpoint with FileCheckpointStorage/Cosmos. Later checkpoints fail to decode because the restricted unpickler trusts agent_framework. and agent_framework_orchestrations. but not agent_framework_declarative. The paused state is never saved, so resume fails with "unknown request ID". I did not run anything; the findings below come from reading the code.

    Recommendation

    Code change warranted. It is a framework defect, the documented/sample paths hit it, and it is the same class as #8258. The workaround (hand-written allowlist of internal module paths) is not adequate. Keep the fix in the declarative package using the public register_checkpoint_type, with the core prefix list unchanged. Registration is also what the orchestrations package does at import time (agent_framework_orchestrations/__init__.py). Core's register_checkpoint_type is process-wide, and decode_checkpoint_value merges registered keys on every restricted decode. So File and Cosmos (which both go through that function) see registrations made after storage creation. Unrelated custom types are still blocked, and dotted names stay blocked.

    Type inventory (from source)

    Sent via send_message (message envelopes), all in _declarative_base.py: ActionTrigger, ActionComplete, ConditionResult, LoopIterationResult, LoopControl. These are plain/dataclass types that get pickled.

    Used with request_info (pending requests and responses):

    • ExternalInputRequest/Response (Question and RequestExternalInput)
    • AgentExternalInputRequest/Response
    • ToolApprovalRequest/Response (function tools and MCP)
    • MCPToolApprovalRequest

    Stored in workflow state: ExternalLoopState (read back via ctx.state.get(EXTERNAL_LOOP_STATE_KEY)). The main declarative state is a TypedDict (a plain dict), so it should not need registration.

    Nested types: Message and Content inside the agent request/response are core agent_framework.*, already trusted. The unpickler resolves nested classes too, so I would confirm nothing else nested slips through.

    Probably not needed, but worth proving by test rather than assuming: AgentResult, ToolInvocationResult, HttpRequestInfo/Result, MCPToolInvocation/Result. I did not find these passed to send_message, request_info or state, so I would not register them. Every extra registration widens the trust surface.

    Open point: ToolApprovalResponse and ExternalInputResponse contain Any fields (value, arguments). Any user-supplied custom type there remains the app's responsibility.

    Proposed plan (nothing changed yet)

    1. In agent_framework_declarative import-time code, register exactly the 12 types above (message envelopes, request/response pairs, MCPToolApprovalRequest, ExternalLoopState), with a comment per group like the orchestrations precedent. Check the import path covers import agent_framework_declarative._workflows use.
    2. Round-trip tests: for each registered type, encode/decode with a restricted decoder (File and Cosmos codec path). Add a negative test that an unregistered declarative helper (for example AgentResult) and an unrelated agent_framework_foo type are still rejected, and that the core prefix tuple is unchanged.
    3. Fresh-workflow pause/resume tests with FileCheckpointStorage and no allowlist: (a) RequestExternalInput/Question, asserting multiple checkpoints and the correct response; (b) function-tool approval; (c) MCP approval; (d) agent externalLoop resume to cover ExternalLoopState. Replace the allowlist workaround in test_workflow_run_kwargs_survive_file_checkpoint_restore.
    4. Fix the WorkflowFactory docstring (path= is not a valid argument) and check the function_tools sample.
    5. Done when the issue's repro writes all checkpoints and prints ['Hello - answer yes'].

    Trade-offs and review of #10076

    Process-wide registration means importing declarative trusts these types for all storages in the process. That is acceptable because they are framework-owned dataclasses, and it is what the orchestration precedent chose. I treated #10076 only as an existing proposal and did not review it. Its inventory should be judged against the list above, and not merged as readiness evidence until the approval/agent-loop resume tests exist. Size is small; the risk is in completeness, not volume.

    Generated by Issue Planning for #10074 · copilot · auto · ~35 AIC · ⌖ ~1 AIC · ⊞ 4.8K · ◷

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

declarativeUsage: [Issues, PRs], Target: declarative agents and workflowspythonUsage: [Issues, PRs], Target: Pythonready-for-implementationMaintainers agree the problem and direction are ready for implementation.reproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflow

Type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions