Skip to content

Python: [Bug]: File checkpoints overwrite case-distinct explicit IDs on Windows #9444

Description

@ktz03

Observed Behavior

On a case-insensitive Windows filesystem, FileCheckpointStorage.save() accepts two distinct explicit string checkpoint IDs that differ only by case. The second save replaces the first file. A later load(first_id) succeeds but returns the second checkpoint's ID, workflow name, graph signature and state. The first workflow's checkpoint listing becomes empty.

For example, both Round-A and round-a save successfully. Before the second save, loading Round-A returns the first state. Afterwards it returns round-a, second-workflow, and {"marker": "second"}. No error is raised.

Expected Behavior

Distinct accepted checkpoint IDs should retain their respective snapshots. If an ID cannot be represented independently by this backend, it should be rejected before an existing different checkpoint is replaced. Please advise on the intended compatibility policy before implementation.

Steps to Reproduce

  1. On a Windows directory with case-insensitive filenames, create a FileCheckpointStorage.
  2. Save two public WorkflowCheckpoint objects with explicit IDs Round-A and round-a and different workflow names/states.
  3. Load the first returned ID and list checkpoints for the first workflow.

Minimal Reproduction

import asyncio
from tempfile import TemporaryDirectory

from agent_framework import FileCheckpointStorage, WorkflowCheckpoint


async def main():
    with TemporaryDirectory() as directory:
        storage = FileCheckpointStorage(directory)
        first = WorkflowCheckpoint(
            workflow_name="first-workflow",
            graph_signature_hash="topology-a",
            checkpoint_id="Round-A",
            state={"marker": "first"},
        )
        second = WorkflowCheckpoint(
            workflow_name="second-workflow",
            graph_signature_hash="topology-b",
            checkpoint_id="round-a",
            state={"marker": "second"},
        )
        first_id = await storage.save(first)
        print("First read before second save:", (await storage.load(first_id)).state)
        second_id = await storage.save(second)
        restored = await storage.load(first_id)
        print("Successful save IDs:", first_id, second_id)
        print("First read after second save:", restored.checkpoint_id, restored.workflow_name, restored.state)
        print("First workflow IDs:", await storage.list_checkpoint_ids(workflow_name="first-workflow"))


asyncio.run(main())

Error Messages and Stack Traces

There is no exception. Actual output:

First read before second save: {'marker': 'first'}
Successful save IDs: Round-A round-a
First read after second save: round-a second-workflow {'marker': 'second'}
First workflow IDs: []

Package Versions

agent-framework-core source version 1.22.0 at main 9a1d66b3e973bdbfdcd4359d72c5f6fd4ff4d931. The existing environment has installed package metadata 1.19.0; the executed Core modules were loaded from and byte-bound to the stated main source. Dependencies were cached, not freshly locked.

Python Version

Python 3.14.3

Operating System

Windows

Regression

Unknown

Additional Context

The same case-distinct pairs are preserved by InMemoryCheckpointStorage. Two different lowercase IDs round-trip separately in both backends. TeamReview / teamreview also reproduce the file-backend overwrite. These are six backend/pair controls, with two file identity failures and four normal controls.

This concerns explicitly supplied string IDs. CheckpointID is a str, and existing checkpoint tests use non-UUID IDs. _validate_file_path currently uses {checkpoint_id}.json directly after a directory containment check. The report does not demonstrate a failure of the default generated UUID IDs, a full workflow resume, other operating systems, concurrency, cloud storage, or a security boundary.

#8181/#8214 concern value encoding/decoding symmetry and listing behavior after decode errors; #7306 protects session-store filenames in another path. This report concerns checkpoint-file identity on successful saves. No product patch is proposed; a portable encoding or rejection policy needs maintainer direction and compatibility consideration.

AI-assisted analysis, reproduction, and writing.

Acknowledgements

  • I searched existing issues and did not find a duplicate.
  • I personally verified this behavior and the reproduction details are authentic.
  • I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.

Activity

  1. added
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Oct 16, 2026
  2. added
    reproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflow
    on Oct 16, 2026
  3. github-actions commented on Oct 16, 2026

    @github-actions
    Contributor

    🤖 Automated triage reproduction notes (agent-authored — trust but verify)

    Agent analysis

    python/packages/core/agent_framework/_workflows/_checkpoint.py::FileCheckpointStorage.save overwrites an existing checkpoint when explicit IDs differ only by case on a case-insensitive filesystem. Minimal repro: save checkpoints named Round-A and round-a with different payloads, then load Round-A; it returns the second checkpoint.

    • Failing test: python/packages/core/tests/workflow/test_checkpoint.py::test_file_checkpoint_storage_preserves_case_distinct_ids
    • Files examined: python/packages/core/agent_framework/_workflows/_checkpoint.py, python/packages/core/tests/workflow/test_checkpoint.py, python/packages/core/pyproject.toml
    • Tests run: test_file_checkpoint_storage_list (passed), test_file_checkpoint_storage_preserves_case_distinct_ids (failed)
    • Reported version: 1.22.0
    • Current version: 1.22.0
  4. added
    workflowsUsage: [Issues, PRs], Target: Workflows
    and removed
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Oct 16, 2026
  5. eavanvalkenburg commented on Nov 3, 2026

    @eavanvalkenburg
    Member

    /plan Consolidate the current-main checkpoint-ID alias data-integrity repair through existing PR #9466 and its maintainer review, not a new competing implementation. Current FileCheckpointStorage maps accepted string IDs directly to filenames and can replace/read the wrong checkpoint on aliasing filesystems. Existing review already establishes a fail-closed exact persisted-ID contract across save/load/delete; do not reopen that as a generic core ownership question or silently rename every existing checkpoint. Use current PR head and distinguish author-reported fixes from validated remaining gaps. Check real storage-directory identity across parent spelling aliases, worker-side destination ownership, persisted-ID verification during load/delete interleavings, path-bearing accepted IDs, NFC/NFD/decomposing filesystems, same-ID updates and independent missing IDs on case-sensitive volumes. Keep scans off the event loop and bound/document costs and process-local versus cross-process guarantees. Reuse current conformance/regression work, require native-platform controls where practical and explicit simulation limits, and identify the smallest remaining compatibility/identity scope before readiness or assignment. Do not repeat prior path-prefix or scan findings as still unfixed without testing the latest head.

  6. eavanvalkenburg commented on Nov 12, 2026

    @eavanvalkenburg
    Member

    /plan Consolidate the current-main checkpoint-ID alias data-integrity repair through existing PR #9466 and its maintainer review, not a new competing implementation. Current FileCheckpointStorage maps accepted string IDs directly to filenames and can replace/read the wrong checkpoint on aliasing filesystems. Existing review already establishes a fail-closed exact persisted-ID contract across save/load/delete; do not reopen that as a generic core ownership question or silently rename every existing checkpoint. Use current PR head and distinguish author-reported fixes from validated remaining gaps. Check real storage-directory identity across parent spelling aliases, worker-side destination ownership, persisted-ID verification during load/delete interleavings, path-bearing accepted IDs, NFC/NFD/decomposing filesystems, same-ID updates and independent missing IDs on case-sensitive volumes. Keep scans off the event loop and bound/document costs and process-local versus cross-process guarantees. Reuse current conformance/regression work, require native-platform controls where practical and explicit simulation limits, and identify the smallest remaining compatibility/identity scope before readiness or assignment. Do not repeat prior path-prefix or scan findings as still unfixed without testing the latest head.

  7. github-actions commented on Nov 12, 2026

    @github-actions
    Contributor

    Summary

    FileCheckpointStorage turns each checkpoint ID straight into a filename ({checkpoint_id}.json). On a filesystem that treats two spellings as one name (case-insensitive NTFS/APFS, Unicode-folding macOS), two different IDs share one file, so a save can silently replace another checkpoint and a load can return the wrong one. This comment is a re-assessment of current main only. It reads the code but runs nothing.

    Limit of this review: this workflow does not let me read the open PR #9466 (diff, head or reviews). I therefore cannot separate "author-reported fixes" from "validated remaining gaps" on the PR head, and I did not repeat any earlier finding as unfixed. The checklist below is for the maintainer and PR author to check against the PR head.

    What I confirmed on main

    • _validate_file_path only checks that storage_path / f"{id}.json" resolves under the storage directory. Nothing guards against aliasing. save, load and delete all use it.
    • The destination ownership queue, _delete_lock striping and hash(file_path) are all keyed on the path spelling. Two spellings that reach the same file get different queues and locks. The delete locks and write queues are also only process-local, so they give no cross-process guarantee.
    • load does not compare the checkpoint_id stored inside the file with the one requested. That is why the report returns the wrong checkpoint without any error.
    • list_checkpoints globs *.json and decodes every file with asyncio.to_thread. The scan is off the event loop, but its cost grows with the number of files and is not documented.
    • The repository already has a solved pattern for this problem. _filesystem._storage_key_segment (see core AGENTS.md) maps IDs injectively. It uses lowercase ASCII IDs verbatim and base32-encodes uppercase and non-ASCII ones. It is used by session, todo and memory stores.

    Recommendation

    Code change: warranted. This is a real data-integrity bug (silent overwrite or wrong read). No workaround exists except "only use lowercase ASCII IDs". Default UUID IDs are unaffected, so the exposure is limited to explicit custom IDs. The fix should go through the existing PR rather than a new implementation. A fail-closed, exact-ID contract is the proportionate choice, and renaming existing files is not needed.

    Why fail-closed rather than re-encoding everything:

    • Existing checkpoints stay readable, because lowercase-safe IDs keep their current file names.
    • A fully injective mapping would change on-disk names for existing mixed-case IDs. That would need a migration or a read fallback.

    Proposed plan (nothing has been changed)

    1. Exact-ID contract. Treat the ID stored inside the file as the source of truth. save refuses to replace a file whose stored ID differs. load and delete verify the stored ID before returning or removing. A mismatch fails with WorkflowCheckpointException. Same-ID updates must still work.

    2. Check the interleavings. Make sure delete cannot remove a file after another save has put a different ID there between the verify and unlink steps. Say in the docs how far this is guaranteed: serialized within one process, best-effort across processes.

    3. Key locks and queues by real identity. Use a canonical form of the storage directory (resolved, with case/Unicode folding where relevant) plus the folded filename. That way Dir and dir, or symlinked parents, share one queue. Destination ownership must be held by the same code path that runs the worker write.

    4. Path-bearing IDs. Reject IDs with separators, .., drive or stream syntax (a:b), trailing dot/space and reserved Windows names, if they are not rejected already. Do not rely on the containment check alone.

    5. Unicode. Decide whether NFC/NFD variants of the same ID count as distinct. The simplest rule is to reject an ID when an existing file with a different stored ID maps to the same name. Do not silently normalize.

    6. Independent IDs on case-sensitive volumes. Round-A and round-a should both work on Linux ext4 unless the collision check says they alias on this particular volume. Avoid a blanket "reject uppercase", which would break valid current use.

    7. Performance. Keep any collision scan in asyncio.to_thread. Prefer a direct stat/open of the target over a full directory scan, and document the cost if a scan is unavoidable.

    8. Tests.

      • Keep the reported case-pair repro as a regression test.
      • Add same-ID overwrite, a missing independent ID, load/delete after an alias save, and path-bearing IDs.
      • Run real-platform controls on Windows and macOS CI where available.
      • Use simulated aliasing (for example a patched resolver) only for the rest, and state in the test that this does not prove native filesystem behavior.
    9. Docs. Update the FileCheckpointStorage docstring with the ID rules, the failure behavior, and the process-local versus cross-process limits.

    Smallest remaining scope to check before "ready"

    Compare the PR head against items 1–3 and 5. Those carry the real risk: persisted-ID verification, canonical-directory lock keys, and the Unicode decision. Items 4, 6 and 7 are likely small additions or confirmations. If the PR head already covers some of these, those items should be marked verified, not reopened.

    Questions for the maintainer: should NFC/NFD-equivalent IDs be rejected on every platform, or only when an actual collision is detected on the volume? Should the contract be documented as best-effort across processes?

    This is a proposal only; no files, branches or PRs were changed.

    Generated by Issue Planning for #9444 · copilot · auto · ~30 AIC · ⌖ ~1 AIC · ⊞ 4.8K · ◷

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

pythonUsage: [Issues, PRs], Target: Pythonready-for-implementationMaintainers agree the problem and direction are ready for implementation.reproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflowworkflowsUsage: [Issues, PRs], Target: Workflows

Type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions