Skip to content

Python: isolate file memory description sidecars per file - #10299

Open
Ali Adnan (AAliKKhan) wants to merge 1 commit into
microsoft:mainfrom
AAliKKhan:fix-file-memory-description-sidecar-collisions
Open

Ali Adnan (AAliKKhan) wants to merge 1 commit into
microsoft:mainfrom
AAliKKhan:fix-file-memory-description-sidecar-collisions

Conversation

@AAliKKhan

Copy link
Copy Markdown

Motivation & Context

FileMemoryProvider stored a memory's description in a companion sidecar derived by stripping the original extension before appending _description.md, so notes.md and notes both mapped to notes_description.md. Any two files whose names differ only by extension therefore shared one sidecar: one file's description overwrote the other's, rewriting either file without a description silently deleted the other's description, and deleting either file removed both descriptions along with both files. The issue's repro (two memory files whose names differ only by extension) loses data on ordinary write/delete sequences.

Description & Review Guide

  • What are the major changes?
    • Sidecars are now named after the whole memory file name: notes.md → notes.md._description.md. The mapping is injective, and the new names are disjoint from every legacy <stem>_description.md name by construction (removing an extension always cuts at a dot, so a stem never ends in .), which satisfies the format-isolation option from the issue without any migration machinery.
    • Sidecars whose name would exceed the 255-byte filesystem name budget fall back to a SHA-256 digest name under the reserved ~memory-desc- prefix. The reserved prefix is matched after NFKC normalization + case folding, so no spelling a case-folding or Unicode-folding filesystem treats as reserved can forge a sidecar name that collides with a digest.
    • Digest sidecars derive from the backend's canonical stored name (via list_children + file_exists, no new public API), so a case-folding store cannot scatter one description across case-aliased spellings of the same file.
    • Sidecars written by the pre-isolation format are still read as a fallback, but only when no other existing memory file shares the stem (stems compared with casefold); an ambiguous sidecar is neither read nor destroyed. Clearing or deleting a file clears its unambiguous legacy sidecar too.
    • InMemoryAgentFileStore.write now preserves the recorded display case when overwriting through a differently-cased path, matching case-insensitive filesystem behavior and keeping canonical-name-derived identities stable.
  • What is the impact of these changes?
    • Fixes the data-loss collision for existing stores going forward; existing unambiguous legacy sidecars keep working unchanged.
    • New sidecar file names appear on disk for descriptions written after this change (<name>._description.md instead of <stem>_description.md).
  • What do you want reviewers to focus on?
    • The format-isolation choice (<full name>._description.md vs. a migration) — flagged for confirmation since the issue allowed either.
    • The legacy-sidecar fallback guards (ambiguity check and case folding) and the reserved-prefix NFKC + casefold matching.

Related Issue

Fixes #9871

AI Assistance

  • No material AI assistance was used.
  • This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

AI assistance details: Implementation, tests, documentation, and this PR description were drafted with AI assistance; I reviewed, understood, and verified the changes and accept responsibility for them.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

The description sidecar stripped the original extension before appending _description.md, so notes.md and notes shared notes_description.md: one file's description overwrote the other's, rewriting one without a description deleted the other's, and deleting either removed both.

Sidecars are now named after the whole memory file name (notes.md -> notes.md._description.md), which is injective and disjoint from every legacy <stem>_description.md name by construction, so an old sidecar can never be read as another memory's description. Names whose sidecar would exceed the 255-byte filesystem name budget fall back to a SHA-256 digest sidecar under the reserved ~memory-desc- prefix; that prefix is matched after NFKC normalization and case folding so no spelling a folding filesystem treats as reserved can forge a digest sidecar. Digests derive from the backend's canonical stored name, so a case-folding store cannot scatter one description across case-aliased spellings.

Legacy sidecars are still read as a fallback, but only when no other existing memory file shares the stem (stems compared with casefold); an ambiguous sidecar is neither read nor destroyed, and clearing or deleting a file clears its unambiguous legacy sidecar.

Fixes microsoft#9871
Copilot AI balanced review requested due to automatic review settings November 12, 2026 13:51
@agent-framework-automation agent-framework-automation Bot added documentation Usage: [Issues, PRs], Target: documentation in the code base and learn docs python Usage: [Issues, PRs], Target: Python labels Nov 12, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved sidecar naming collisions, filename alias handling, and overlong legacy paths can overwrite or orphan descriptions and cause listing or deletion failures.

4 open issues
What changed in this PR

Updates Python’s FileMemoryProvider to address description-sidecar collisions between memory files whose names differ only by extension, as reported in #9871.

Changes:

  • Derives sidecar names from full filenames, with digest fallback for long or reserved names.
  • Adds guarded legacy-sidecar fallback and preserves stored filename casing during in-memory overwrites.
  • Adds naming and lifecycle regression tests and updates contributor documentation.
File Description
python/​packages/​core/​tests/​core/​test_harness_file_memory.py Tests sidecar isolation, legacy compatibility, long names, and case aliases.
python/​packages/​core/​AGENTS.md Documents sidecar naming and compatibility behavior.
python/​packages/​core/​agent_framework/​_harness/​_file_memory.py Implements new naming, canonical-name resolution, and legacy fallback and cleanup.
python/​packages/​core/​agent_framework/​_harness/​_file_access.py Preserves recorded filename casing when overwriting in-memory files.

Review effort: Balanced

# always cuts at a dot). So ``<name>._description.md`` can never equal any
# legacy sidecar name, which makes the two namespaces disjoint by construction:
# an old sidecar can never be read as another memory's description.
_NEW_DESCRIPTION_SUFFIX = f".{_DESCRIPTION_SUFFIX}"
Comment on lines +231 to +235
lowered = file_name.lower()
for entry in entries:
if entry.name.lower() == lowered:
return entry.name
return file_name
Comment on lines +467 to +473
Digest names are derived from the backend's canonical stored file name
so a case-folding store cannot scatter one description across several
sidecars; verbatim names need no resolution because the store folds them
on its own.
"""
if _requires_digest_sidecar(file_name):
file_name = await _canonical_file_name(self.store, working_folder, file_name)
"""Return the legacy sidecar path, or ``None`` when it is not unambiguously ``file_name``'s."""
if not _legacy_sidecar_is_unambiguous(file_name, existing):
return None
return _combine_paths(working_folder, _legacy_description_file_name(file_name))
@AAliKKhan

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

This branch was successfully deployed

1 active deployment
github-app-auth — 59fe558b Deployed Nov 13, 2026 by AAliKKhan via team_check #9420
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Usage: [Issues, PRs], Target: documentation in the code base and learn docs python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python: [Bug]: file memory description sidecars collide for names differing by extension

2 participants