Skip to content

Python: give memory topic files an injective filename stem - #9724

Open
Manohar Paturi (ManoharPaturi) wants to merge 8 commits into
microsoft:mainfrom
ManoharPaturi:fix/memory-topic-slug-collision
Open

Manohar Paturi (ManoharPaturi) wants to merge 8 commits into
microsoft:mainfrom
ManoharPaturi:fix/memory-topic-slug-collision

Conversation

@ManoharPaturi

@ManoharPaturi Manohar Paturi (ManoharPaturi) commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Motivation & Context

MemoryFileStore keyed every topic file by _slugify_topic(topic), which is many to one: C ++ and C # both slug to c. write_topic overwrites the slug path unconditionally, so writing a second topic that shares a slug silently destroyed the first topic's memories. Reproduced on main: after writing both topics, get_topic("C ++") returned the C # memory and the C ++ memory was gone. The store's own path components for owner, session, source, and scope all use the injective _storage_key_segment derivation for exactly this reason; the topic filename was the one mapping left below that standard.

Description & Review Guide

  • What are the major changes? The topic filename stem is now the readable slug plus an 8 character digest of the normalized topic, so distinct normalized topics never share a file. Writes path by the record's normalized topic, and reads resolve the digest stem first.
  • What is the impact of these changes? Topics that share a slug coexist with their own memories. Reads keep working for stores written before this change through two fallbacks: the legacy slug-only filename, then a slug-prefix scan that also serves lookups made by slug (several tests look topics up by record.slug). When a slug-only lookup is ambiguous because several topics share that slug, the file whose recorded topic matches the lookup wins, else the first stem in sorted order keeps the choice deterministic.
  • What do you want reviewers to focus on? The fallback chain in _existing_topic_path, and the regression test writing C ++ then C # and reading both back, which fails on main.

Related Issue

Fixes #9723

AI Assistance

  • No material AI assistance was used.
  • This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Manohar Paturi (@ManoharPaturi) Four material issues remain in the memory topic storage changes:

  1. In python/packages/core/agent_framework/_harness/_memory.py:811-825, glob(f"{prefix}*.md") accepts longer unrelated slugs and returns the first match. A missing topic can therefore read, merge into, or delete another topic; colliding names such as C++ and C# can also merge. Please validate exact topic/slug identity rather than accepting arbitrary prefixes.
  2. In _memory.py:884-894, updating a legacy filename writes a new digest file without retiring the matching legacy file. Both remain visible, lookup can return stale content, and deletion can resurrect the legacy record. Please implement a one-authoritative-record migration/update lifecycle.
  3. In _memory.py:783-784, index links and provider selection still use the old nonunique slug instead of the actual persisted filename/exact topic identity. Collision-safe files can therefore produce broken links and duplicate the wrong topic during injection. Please use the persisted storage identity consistently.
  4. In _memory.py:886, writing from record.topic ignores the public MemoryTopicRecord(slug=...) override. Please preserve explicit slug lookup compatibility while introducing collision-safe storage identities.

Please address all four before merge.

Distinct topics can share a readable slug, so a slug-only stem let the second topic silently overwrite the first. The stem now appends a digest of the normalized topic, and the storage layer keeps one authoritative record: lookups validate every candidate against the record it actually holds instead of accepting a filename prefix, so a missing topic can never read, merge into, or delete a longer slug; rewriting a topic that lived at a legacy slug-only file retires the legacy file; index pointer links and provider lookups use the persisted stem; and an explicit slug override is persisted in the record and stays findable by that slug, without letting lookups for other names bleed into the overridden record.

Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
@ManoharPaturi

Copy link
Copy Markdown
Contributor Author

All four addressed in 8d4eb01, verified end to end against the store before pushing.

  1. Exact identity: _existing_topic_path no longer accepts a filename prefix. Every glob candidate is read and validated against the record it actually holds, first by normalized topic, and a missing topic such as coll under Collection now raises FileNotFoundError instead of reading the longer slug. Colliding C++ and C# each resolve to their own record, and looking up a bare C that matches neither record finds nothing rather than an arbitrary same-slug file.
  2. One authoritative record: write_topic resolves the previous path for the topic before writing and retires it when it differs, so rewriting a topic that lived at a legacy slug-only file leaves exactly the digest-stemmed file. A test seeds a legacy file, rewrites, and asserts the legacy file is gone and only the new stem remains.
  3. Persisted identity everywhere: index entries are built from record.storage_stem, so every pointer line links a file that exists, with a test asserting all links resolve when two topics share a slug, and injection consumes those same entries.
  4. Slug override: the explicit slug is persisted in the record file and used for the storage stem, so get_topic by that slug resolves the overridden record after a reload, while lookups for other names cannot bleed into it (a guard test asserts machine does not find the ml-notes record).

The five new tests fail on the previous head. The memory suite passes at 40 tests, ruff check and format are clean.

@ManoharPaturi

Copy link
Copy Markdown
Contributor Author

Eduard van Valkenburg (@eavanvalkenburg) the PR body now carries the required Contribution Checklist with the non-breaking item confirmed, matching the template. Everything from your review is addressed in the latest push and the thread replies have the verification details.

@eavanvalkenburg

Copy link
Copy Markdown
Member

Manohar Paturi (@ManoharPaturi) Two additional cases at 8d4eb109 need attention:

  1. In python/packages/core/agent_framework/_harness/_memory.py:854-857, default-slug reloads now fail. A normally written Travel Preferences record still reports slug="travel-preferences", but reloading through get_topic(..., record.slug) raises FileNotFoundError. Consolidation still reloads by that slug at line 1666 and catches the exception, silently omitting the topic; the forced consolidation probe processes 1 topic on base and 0 on this head. Please preserve unambiguous default-slug compatibility or make consolidation use the exact topic/persisted identity, without restoring ambiguous prefix matching.

  2. In _memory.py:143-146, the 8-hex-character digest still permits silent overwrite. Both 主题乚丢一 and 主题亊不七 become memory-topic-74043f1d.md. Writing them with different memories leaves one record, and reading the first topic returns the second topic's fact. Please use a collision-resistant storage identity and validate the existing record's topic before replacing it, with a regression for this pair.

The topic digest widens to 64 bits and write_topic verifies the occupant
of an existing stem before reusing it, extending the digest when another
topic already lives there. Slug lookups answer a single unambiguous
holder again so default slug reloads work, and consolidation reloads each
topic by exact identity instead of by slug.

Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
@ManoharPaturi

Manohar Paturi (ManoharPaturi) commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor Author

Both addressed in cef2b8e, with your two probes reproduced on the previous head first.

  1. Default slug reloads work again. A slug lookup now answers when exactly one record claims the slug, so Travel Preferences reloads through get_topic(record.slug). The rule only fires when the lookup string is itself a slug, because a topic name that merely slugifies into the family must not borrow a neighbor's file: your two CJK topics both slugify to memory-topic, and my first cut of the slug rule let the second write retire the first topic exactly that way, which is why the guard also checks the lookup shape. Consolidation reloads by record.topic now regardless, so the forced probe reaches every topic through exact identity. The regression drives the consolidate tool end to end with a Travel Preferences topic and asserts it comes back consolidated; on the previous head the probe skips it and the test fails.

  2. Digest collisions can no longer silently overwrite. The stem digest widened to 64 bits, so your pair separates at sixteen hex characters, and write_topic now reads the occupant of an existing stem before reusing it, extending the digest when a different topic already lives there. One regression writes your exact pair and asserts both files survive with isolated reads, and a second seeds one topic's record at another topic's primary stem to prove the write side extends without depending on hash luck. Records also carry the stem they were read from or written to, so index pointer lines keep naming files that exist after an extension.

The four new tests fail on the previous head and pass now. The memory suite passes at 44 tests, the neighboring harness suites pass at 366, and ruff check and format are clean.

One correction to the verification note above: ruff and the suites were clean, but the same push tripped Package Checks, which runs pyright over the package and rejected the store assigning the record's private stem field from outside its class. That is fixed in f1fd7fa with the stamp routed through a setter, and pyright now reports the file clean, so the earlier "clean" claim should be read as ruff only, not the full check suite.

Package Checks flagged the store assigning the record's private on disk stem field from outside the class. The stamp now goes through a small setter so the assignment happens inside the record.

Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
@ManoharPaturi

Copy link
Copy Markdown
Contributor Author

The Package Checks failure was pyright flagging the store stamping the record's private on disk stem field from outside its class, three sites. Fixed in f1fd7fa: the stamp now goes through a small setter so the assignment happens inside the record. Verified with pyright against the core package config, where the file now reports clean, and the memory suite still passes at 44 tests. The merge gatekeeper was only mirroring that failure.

# 1. The digest-stemmed candidates for this name: the exact files
# write_topic uses, each validated against the record it holds so a
# digest collision with another topic can never answer the lookup.
for candidate in _topic_stem_candidates(topic):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Candidate stems are probed without first bounding their encoded filename length. For a sufficiently long readable slug, the longest candidate plus .md exceeds the filesystem component limit, so Path.exists() raises ENAMETOOLONG before lookup can reach a valid persisted or legacy filename. This can make both existing long topics and newly written topics unreadable, unwritable, or undeletable. Please bound generated candidates before touching the filesystem and resolve valid persisted/legacy stems without first probing oversized names. Add regressions for a long legacy-topic lookup and index-stem reload.

Copilot AI balanced review requested due to automatic review settings November 3, 2026 04:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@ManoharPaturi

Copy link
Copy Markdown
Contributor Author

Fixed in 0a7bbf5, with the failure reproduced on the previous head first.

The write path is where it bites hardest: on the previous head, writing a topic whose slug makes the primary candidate plus .md overshoot the component limit raises ENAMETOOLONG (errno 63 here) before any valid name is reached, so the topic is unwritable. Generated candidates now cap the readable slug part so the longest candidate, digest and extension included, stays within 255 bytes, and every filesystem probe skips oversized names instead of raising, so a valid persisted or legacy stem is still reached.

Regressions cover both sides: a legacy slug only file at 240 characters stays readable, and a topic with a ~470 character slug writes, reloads by topic and by its index stem, with the persisted stem asserted within the limit and every index link resolving. The write regression fails on the previous head with ENAMETOOLONG and passes now; the memory suite is at 46 tests, and pyright reports the file clean.

@eavanvalkenburg

Copy link
Copy Markdown
Member

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MAF Automated Review — Iteration 1

Result: Findings reported
Scope: full PR (4 commit(s)): 8d4eb109881a, cef2b8e95aac, f1fd7fa1b0f1, 0a7bbf53decc
Model: gpt-5.6-sol

Overview

The PR replaces non-injective slug-only filenames with bounded digest stems, validates digest occupants, preserves index links to persisted stems, and adds broad collision and compatibility coverage. Those guards address fresh same-slug collisions and overlong filenames, but the unchecked exact-file fallback can still delete an unrelated legacy topic during migration. Consolidation also regresses for the newly persisted explicit-slug records because it reloads them through the default topic-derived filename family.

Reviewed the supplied pull-request change set across correctness, security/reliability, architecture, and failure behavior.
2 verified findings remained after source verification (1 high, 1 medium) across 1 file. Details are attached to the affected lines below.

Affected areas: python/packages/core/agent_framework/_harness/_memory.py

else:
exact_path = topics_dir / f"{legacy_name}.md"
if exact_path.exists():
return exact_path

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This exact-file fallback does not verify which topic the file contains. On an upgraded store where legacy c.md holds C++, writing C# resolves that unrelated file as previous_path and unlinks it after creating the new C# file; deletion has the same cross-topic effect. Please preserve persisted-stem reads, but require destructive write/delete resolution to verify that a slug-only occupant has the requested normalized topic before it can be retired or removed.

# Reload by the exact topic identity: a slug can be shared
# by several topics, and dropping one of them here would
# silently skip its consolidation.
current_record = self.store.get_topic(session, source_id=self.source_id, topic=record.topic)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An explicit-slug record such as topic Machine Learning with slug ml-notes is stored under ml-notes-<digest>.md, but reloading by record.topic searches the machine-learning-* family. Consolidation therefore catches FileNotFoundError and silently skips every such record, whereas the previous slug-based reload found it. Please reload through the known persisted stem (or otherwise resolve that exact file) while retaining identity-safe behavior for colliding default slugs.

@eavanvalkenburg

Copy link
Copy Markdown
Member

Manohar Paturi (@ManoharPaturi) The new filename scheme still has two topic-identity gaps. First, write_topic uses an alias-capable lookup for the previous record without verifying the record’s actual topic, so writing C++ followed by c—or API followed by api—can delete or merge the other topic despite distinct digest filenames. Second, consolidation reloads by record.topic, but lookup searches only the filename family derived from that topic, so a record stored under a custom slug can be skipped and later duplicated. Please separate exact-topic resolution from explicit slug aliases, validate topic ownership before merge/retirement, and use one canonical identity for internal reload and cleanup. Add lifecycle and consolidation regressions for both cases.

@eavanvalkenburg

Copy link
Copy Markdown
Member

Manohar Paturi (@ManoharPaturi) Two additional correctness issues remain at 0a7bbf53:

  1. In python/packages/core/agent_framework/_harness/_memory.py:354, persisted-stem aliases bypass the consolidation lock. A write through an index slug locks the digest-bearing stem while consolidation locks the readable topic slug. I reproduced an append that succeeds during the consolidation client call and is then overwritten by the stale consolidation result; the same operation waits and survives on base. Please resolve supported aliases to one canonical topic-lock identity, reload under that lock, and add a paused-consolidation regression proving the acknowledged write survives.

  2. In _memory.py:157, truncation immediately after a separator can produce a filename with --<digest>, while index construction and stem lookup normalize it to -<digest>. For topic = "a" * 186 + " " + "b" * 10, the index therefore names a nonexistent file and selected-topic loading raises FileNotFoundError. Please keep the bounded slug canonical and preserve the exact persisted-stem identity, with a regression for a truncation boundary immediately after a separator.

slug=record.slug,
# The persisted stem, not the readable slug: distinct topics can
# share a slug, and an index link must point at the real file.
slug=record.storage_stem,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Manohar Paturi (@ManoharPaturi) Long topic names can produce broken index links — The new digest suffix can push an index line past the 150-character limit. The current trimming then cuts the link destination itself, removing the .md suffix and closing parenthesis, so the index no longer opens the topic. Please trim the summary or visible label while keeping the full filename, and add a regression for a long topic name.

…d cleanup

Locks key on the persisted stem so writes through any alias serialize with consolidation, consolidation locks and reloads by the same stem, retirement and deletion verify the occupant topic before unlinking, the bounded slug stays canonical after a separator boundary, and index trimming never cuts the link destination. Regressions cover the paused consolidation append, the ownership guard, the truncation boundary, and long index links.

Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
@ManoharPaturi

Copy link
Copy Markdown
Contributor Author

All five open items across your last two rounds and the automated review are addressed in b87b00c, each reproduced on the previous head before fixing.

  1. One lock identity. _topic_async_lock now resolves the topic to its persisted stem and keys the lock on that, so a write through an index slug, the readable slug, or the topic name all serialize against consolidation on the same lock. The paused consolidation regression holds the consolidation client call, appends through the tool during the pause, and asserts the acknowledged write survives the consolidation write afterwards; on the previous head it is overwritten.

  2. Canonical bounded slug. Truncation landing right after a separator strips the trailing dash, so the stem never carries a double dash, and the regression drives your exact "a" times 186 plus separator plus "b" times 10 shape through write, index build, and stem reload; it fails on the previous head where the index names a file that does not exist.

  3. Ownership before retirement and deletion. A new _path_holds_topic check reads the occupant of any alias capable resolution, and both write_topic retirement and delete_topic refuse to unlink a file that belongs to a different topic. The lifecycle regression seeds a legacy c.md holding C++ and writes C# through the shared slug: the C++ record survives; on the previous head it is deleted.

  4. Consolidation reloads by the persisted stem. Records stored under an explicit slug now reload through the one name that resolves their exact file, and the lock uses the same stem, so no record shape is skipped and later duplicated by a stale write; the regression consolidates a Machine Learning record stored under ml-notes end to end.

  5. Index trimming keeps the destination. Trimming now shortens the summary first, then the visible label, and when even the bare link busts the 150 character budget the line is kept whole rather than emitting a cut destination. The regression builds an index entry whose digest stem alone overflows the budget and asserts the pointer line still opens the file.

The memory suite passes at 52 tests, ruff and pyright are clean on the file.

@eavanvalkenburg

Copy link
Copy Markdown
Member

Manohar Paturi (@ManoharPaturi), I found two material regressions on the current head:

  • python/packages/core/agent_framework/_harness/_memory.py:1036: get_topic accepts readable and digest-bearing slugs, but delete_topic rejects those valid aliases because it compares the alias directly with the topic heading. Please preserve deletion by every documented lookup alias and add coverage.
  • python/packages/core/agent_framework/_harness/_memory.py:1798: consolidation passes storage_stem to MemoryStore.get_topic; non-file stores implementing the documented topic/slug contract can then skip every record. Please keep consolidation compatible with conforming non-file stores and add a custom-store regression.

Deletion rejected the readable slug and persisted stem that get_topic answers, and consolidation reloaded by the file-specific stem, which made conforming non-file stores skip every record. Deletion now accepts every documented alias through the same resolution get_topic uses, and consolidation reloads by the persisted stem first with the topic name and slug as documented fallbacks, so a custom store resolves by topic. The unused pointer trim helper that tripped the package checks is removed and the paused consolidation test is fully typed.

Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
@ManoharPaturi

Copy link
Copy Markdown
Contributor Author

Both regressions are fixed in cbdc13d, each reproduced on the previous head before touching anything.

Deletion accepts every documented alias. The old guard compared the lookup against the topic heading, so get_topic answered the readable slug and the persisted stem while delete_topic refused both; your probe now deletes through all three and the regression rewrites the topic and deletes by each alias in turn, failing on the previous head at the first slug attempt. The guard itself was doing the write path's job: the lookup resolution already rejects an ambiguous slug and validates digest stems, so the file it names is the record the lookup resolves to, and write_topic retirement keeps the topic ownership check where a resolution can genuinely name another topic's file.

Consolidation no longer depends on the file-specific stem. The reload tries the persisted stem first (the exact identity for every file store shape, including explicit slug records whose slug another topic shares), then the topic name and the readable slug, both part of the documented get_topic contract. A conforming custom store whose get_topic resolves by topic or slug answers through the topic alias even though its records carry no stem, so no record is skipped. The regression runs a minimal dict backed MemoryStore through forced consolidation end to end and asserts the record comes back consolidated; on the previous head the count is zero.

The three red checks were also mine, all traced and fixed: the package and dependency bounds failures were the now unused pointer trim helper left behind by the index trimming rework, removed here, and the typing check failures were the paused consolidation test's stub, now fully typed with the real signatures. Pyright reports the source file and the test file clean, the memory suite passes at 54 tests, the neighboring harness suites pass at 372, and ruff check and format are clean.

because a legacy heading may spell the topic differently than a lookup.
"""
record = MemoryTopicRecord.from_markdown(path.read_text(encoding="utf-8"), fallback_topic=path.stem)
return _normalize_topic(record.topic).casefold() == _normalize_topic(topic).casefold()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Manohar Paturi (@ManoharPaturi), writing the slug-shaped topic api after API still resolves the existing API file as an alias, then treats the topics as the same through case-folded ownership comparison and deletes the original file. Please keep case-distinct topics separate and add API/api regressions in both write orders.

slug=record.slug,
# The persisted stem, not the readable slug: distinct topics can
# share a slug, and an index link must point at the real file.
slug=record.storage_stem,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Manohar Paturi (@ManoharPaturi), using the computed storage stem as the public index slug breaks custom MemoryStore implementations: before_run later passes this value to get_topic, while external stores commonly resolve the topic name or readable slug, not a file-store stem. Please keep the public lookup contract stable or use the same topic/slug/stem fallback as reload, with a custom-store before_run regression.

The case folded ownership check let the slug shaped topic api retire the API record through their shared resolution, and the index entry factory put the file store stem in the public slug field that custom stores resolve. Ownership now compares exact normalized topics, the index entry carries the readable slug with the persisted stem in its own link-only field, and injection resolves through the stem, slug, and topic ladder, so conforming custom stores keep working. Regressions cover both case orders and a custom store end to end through before_run.

Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
@ManoharPaturi

Copy link
Copy Markdown
Contributor Author

Both findings are fixed in 35c8ddf, each reproduced on the previous head first.

Case distinct topics stay distinct. The case fold in the ownership check was mine, added for legacy headings spelled differently, and your API after api probe exposed what it really did: the slug shaped lookup resolved the API file as the unambiguous slug holder, the folded comparison called them the same topic, and the write retired the original file outright, not just merged it. The comparison is exact on normalized topics now, which errs toward not retiring, and the regression writes both orders and asserts two files survive with isolated reads; it fails on the previous head with get_topic raising for the first topic written.

The index slug contract is stable again. The entry factory puts the readable slug back in the public field, and the persisted stem rides in its own link-only field: pointer lines link the stem when one is present and the slug otherwise, so distinct topics still get distinct destinations while external consumers resolving by slug or topic keep working. Injection resolves selected entries through the same ladder the consolidation reload uses, stem first then slug then topic, so a custom store answers through either documented alias. The regression is a dict backed store whose entries come from the shared factory driven through before_run end to end, asserting the topic content reaches the injected messages; on the previous head the factory filled the slug with the stem, the store could not resolve it, and before_run failed.

The memory suite passes at 56 tests, pyright reports the source file clean again, and ruff check and format are clean.

else:
exact_path = topics_dir / f"{legacy_name}.md"
if exact_path.exists():
return exact_path

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Manohar Paturi (@ManoharPaturi), this exact-file fallback still affects reads, not only destructive cleanup. In an upgraded store where c.md contains C++, get_topic("C#") returns that record. The provider's write_memory path then merges the C# fact into C++ and writes it under the C++ digest stem, so no C# topic is created. The new ownership guard only prevents unlinking the unrelated file; it does not prevent this cross-topic read/merge. Please verify legacy exact-file ownership before returning it, and add a provider-level _merge_memory regression for two topics with the same legacy slug.

# A resolution through a legacy slug-only file or a shared slug can
# name another topic's record; retiring it would silently destroy
# that topic, so only a file holding this very topic is retired.
previous_path.unlink(missing_ok=True)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Manohar Paturi (@ManoharPaturi), this migration is write-then-unlink without recovery. If the unlink fails or the process stops between those operations, both legacy and digest files remain. list_topics() then returns duplicates, delete_topic() removes only the digest file and the stale legacy topic returns, and later writes never retire it because lookup already selects the digest path. Please make the dual-file state reconcile safely and add regressions for coexistence, unlink failure, and delete resurrection.

entries = store.rebuild_index(session, source_id="src", line_limit=10, line_length=200)
for entry in entries:
assert (store._get_topics_directory(session, source_id="src") / f"{entry.storage_stem}.md").exists()
assert store.get_topic(session, source_id="src", topic=entry.storage_stem).memories == ["kept"]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Manohar Paturi (@ManoharPaturi), the current Test Typing Checks (3.11) job fails mypy, pyright, and ty on tests added in this file. Here entry.storage_stem is str | None; the same job reports additional current-head errors at lines 1527, 1534, 1590, 1596, and 1659. Please make the test doubles and assignments satisfy the core typing contracts so the required typing job passes.

@ManoharPaturi

Copy link
Copy Markdown
Contributor Author

Both findings reproduced on the previous head first, then fixed together with the typing job.

The legacy exact file no longer answers another topic's lookup. Reading C# through the legacy c.md holding C++ returned the C++ record on the previous head, and your provider probe merged the C# fact into C++ with no C# topic ever created; that exact run is now the regression. The exact match splits by shape: a digest bearing stem is an injective identity and answers as the file it names, while a slug only file must hold the lookup's topic (folded for case, because the pre digest scheme stored one file per slug, so a heading spelled with different case is still that one topic). A slug lookup now collects every claiming file and answers only when they all hold one and the same topic, preferring the digest bearing claimant: default slug reloads keep working, and colliding topics stay apart.

The migration now reconciles instead of leaving the pair behind. write_topic no longer retires the file the lookup happens to name, because once the digest file exists the lookup never selects the stale copy again; instead it sweeps every other file whose record holds this topic and unlinks those, so a migration whose unlink failed or never ran heals on the next write. delete_topic resolves the record, then removes every file holding that record's topic, so nothing resurrects. list_topics hides a stale copy of a topic that already lives on a digest stem rather than list the topic twice. Three regressions cover the coexistence listing, the simulated unlink failure that heals on the next write, and the delete that leaves no file behind; all three fail on the previous head.

The typing job failed on my tests and the failures were real. I reproduced all five checkers locally through the workspace test typing task and fixed the tests against their contracts: the storage stem is narrowed before the lookup call, the paused consolidation task is typed to what _merge_memory actually returns, the custom store's search_transcripts and write_state now match the abstract signatures exactly, and the consolidation client is the same harness client the other tests use instead of a loosely typed duplicate. All five checkers pass on the test tree, the strict source pyright reports clean, the memory suite is at 60 with the four new regressions, and the neighboring harness suites pass at 557.

This branch was successfully deployed

1 active deployment
github-app-auth — 58a7ae00 Deployed Nov 11, 2026 by ManoharPaturi via add_label #27604
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

.NET: Python: MemoryFileStore topic slug collisions silently overwrite another topic's memories

3 participants