Repository navigation
Python: give memory topic files an injective filename stem - #9724
Manohar Paturi (ManoharPaturi) wants to merge 8 commits into
Conversation
Eduard van Valkenburg (eavanvalkenburg)
left a comment
There was a problem hiding this comment.
Manohar Paturi (@ManoharPaturi) Four material issues remain in the memory topic storage changes:
- In
python/packages/core/agent_framework/_harness/_memory.py:811-825,glob(f"{prefix}*.md")accepts longer unrelated slugs and returns the first match. A missing topic can therefore read, merge into, or delete another topic; colliding names such asC++andC#can also merge. Please validate exact topic/slug identity rather than accepting arbitrary prefixes. - In
_memory.py:884-894, updating a legacy filename writes a new digest file without retiring the matching legacy file. Both remain visible, lookup can return stale content, and deletion can resurrect the legacy record. Please implement a one-authoritative-record migration/update lifecycle. - In
_memory.py:783-784, index links and provider selection still use the old nonunique slug instead of the actual persisted filename/exact topic identity. Collision-safe files can therefore produce broken links and duplicate the wrong topic during injection. Please use the persisted storage identity consistently. - In
_memory.py:886, writing fromrecord.topicignores the publicMemoryTopicRecord(slug=...)override. Please preserve explicit slug lookup compatibility while introducing collision-safe storage identities.
Please address all four before merge.
Distinct topics can share a readable slug, so a slug-only stem let the second topic silently overwrite the first. The stem now appends a digest of the normalized topic, and the storage layer keeps one authoritative record: lookups validate every candidate against the record it actually holds instead of accepting a filename prefix, so a missing topic can never read, merge into, or delete a longer slug; rewriting a topic that lived at a legacy slug-only file retires the legacy file; index pointer links and provider lookups use the persisted stem; and an explicit slug override is persisted in the record and stays findable by that slug, without letting lookups for other names bleed into the overridden record. Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
ed16818 to
8d4eb10
Compare
|
All four addressed in 8d4eb01, verified end to end against the store before pushing.
The five new tests fail on the previous head. The memory suite passes at 40 tests, ruff check and format are clean. |
|
Eduard van Valkenburg (@eavanvalkenburg) the PR body now carries the required Contribution Checklist with the non-breaking item confirmed, matching the template. Everything from your review is addressed in the latest push and the thread replies have the verification details. |
|
Manohar Paturi (@ManoharPaturi) Two additional cases at
|
The topic digest widens to 64 bits and write_topic verifies the occupant of an existing stem before reusing it, extending the digest when another topic already lives there. Slug lookups answer a single unambiguous holder again so default slug reloads work, and consolidation reloads each topic by exact identity instead of by slug. Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
|
Both addressed in cef2b8e, with your two probes reproduced on the previous head first.
The four new tests fail on the previous head and pass now. The memory suite passes at 44 tests, the neighboring harness suites pass at 366, and ruff check and format are clean. One correction to the verification note above: ruff and the suites were clean, but the same push tripped Package Checks, which runs pyright over the package and rejected the store assigning the record's private stem field from outside its class. That is fixed in f1fd7fa with the stamp routed through a setter, and pyright now reports the file clean, so the earlier "clean" claim should be read as ruff only, not the full check suite. |
Package Checks flagged the store assigning the record's private on disk stem field from outside the class. The stamp now goes through a small setter so the assignment happens inside the record. Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
|
The Package Checks failure was pyright flagging the store stamping the record's private on disk stem field from outside its class, three sites. Fixed in f1fd7fa: the stamp now goes through a small setter so the assignment happens inside the record. Verified with pyright against the core package config, where the file now reports clean, and the memory suite still passes at 44 tests. The merge gatekeeper was only mirroring that failure. |
| # 1. The digest-stemmed candidates for this name: the exact files | ||
| # write_topic uses, each validated against the record it holds so a | ||
| # digest collision with another topic can never answer the lookup. | ||
| for candidate in _topic_stem_candidates(topic): |
There was a problem hiding this comment.
Candidate stems are probed without first bounding their encoded filename length. For a sufficiently long readable slug, the longest candidate plus .md exceeds the filesystem component limit, so Path.exists() raises ENAMETOOLONG before lookup can reach a valid persisted or legacy filename. This can make both existing long topics and newly written topics unreadable, unwritable, or undeletable. Please bound generated candidates before touching the filesystem and resolve valid persisted/legacy stems without first probing oversized names. Add regressions for a long legacy-topic lookup and index-stem reload.
|
Fixed in 0a7bbf5, with the failure reproduced on the previous head first. The write path is where it bites hardest: on the previous head, writing a topic whose slug makes the primary candidate plus Regressions cover both sides: a legacy slug only file at 240 characters stays readable, and a topic with a ~470 character slug writes, reloads by topic and by its index stem, with the persisted stem asserted within the limit and every index link resolving. The write regression fails on the previous head with ENAMETOOLONG and passes now; the memory suite is at 46 tests, and pyright reports the file clean. |
|
/review |
There was a problem hiding this comment.
MAF Automated Review — Iteration 1
Result: Findings reported
Scope: full PR (4 commit(s)): 8d4eb109881a, cef2b8e95aac, f1fd7fa1b0f1, 0a7bbf53decc
Model: gpt-5.6-sol
Overview
The PR replaces non-injective slug-only filenames with bounded digest stems, validates digest occupants, preserves index links to persisted stems, and adds broad collision and compatibility coverage. Those guards address fresh same-slug collisions and overlong filenames, but the unchecked exact-file fallback can still delete an unrelated legacy topic during migration. Consolidation also regresses for the newly persisted explicit-slug records because it reloads them through the default topic-derived filename family.
Reviewed the supplied pull-request change set across correctness, security/reliability, architecture, and failure behavior.
2 verified findings remained after source verification (1 high, 1 medium) across 1 file. Details are attached to the affected lines below.
Affected areas: python/packages/core/agent_framework/_harness/_memory.py
| else: | ||
| exact_path = topics_dir / f"{legacy_name}.md" | ||
| if exact_path.exists(): | ||
| return exact_path |
There was a problem hiding this comment.
This exact-file fallback does not verify which topic the file contains. On an upgraded store where legacy c.md holds C++, writing C# resolves that unrelated file as previous_path and unlinks it after creating the new C# file; deletion has the same cross-topic effect. Please preserve persisted-stem reads, but require destructive write/delete resolution to verify that a slug-only occupant has the requested normalized topic before it can be retired or removed.
| # Reload by the exact topic identity: a slug can be shared | ||
| # by several topics, and dropping one of them here would | ||
| # silently skip its consolidation. | ||
| current_record = self.store.get_topic(session, source_id=self.source_id, topic=record.topic) |
There was a problem hiding this comment.
An explicit-slug record such as topic Machine Learning with slug ml-notes is stored under ml-notes-<digest>.md, but reloading by record.topic searches the machine-learning-* family. Consolidation therefore catches FileNotFoundError and silently skips every such record, whereas the previous slug-based reload found it. Please reload through the known persisted stem (or otherwise resolve that exact file) while retaining identity-safe behavior for colliding default slugs.
|
Manohar Paturi (@ManoharPaturi) The new filename scheme still has two topic-identity gaps. First, |
|
Manohar Paturi (@ManoharPaturi) Two additional correctness issues remain at
|
| slug=record.slug, | ||
| # The persisted stem, not the readable slug: distinct topics can | ||
| # share a slug, and an index link must point at the real file. | ||
| slug=record.storage_stem, |
There was a problem hiding this comment.
Manohar Paturi (@ManoharPaturi) Long topic names can produce broken index links — The new digest suffix can push an index line past the 150-character limit. The current trimming then cuts the link destination itself, removing the .md suffix and closing parenthesis, so the index no longer opens the topic. Please trim the summary or visible label while keeping the full filename, and add a regression for a long topic name.
…d cleanup Locks key on the persisted stem so writes through any alias serialize with consolidation, consolidation locks and reloads by the same stem, retirement and deletion verify the occupant topic before unlinking, the bounded slug stays canonical after a separator boundary, and index trimming never cuts the link destination. Regressions cover the paused consolidation append, the ownership guard, the truncation boundary, and long index links. Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
|
All five open items across your last two rounds and the automated review are addressed in b87b00c, each reproduced on the previous head before fixing.
The memory suite passes at 52 tests, ruff and pyright are clean on the file. |
|
Manohar Paturi (@ManoharPaturi), I found two material regressions on the current head:
|
Deletion rejected the readable slug and persisted stem that get_topic answers, and consolidation reloaded by the file-specific stem, which made conforming non-file stores skip every record. Deletion now accepts every documented alias through the same resolution get_topic uses, and consolidation reloads by the persisted stem first with the topic name and slug as documented fallbacks, so a custom store resolves by topic. The unused pointer trim helper that tripped the package checks is removed and the paused consolidation test is fully typed. Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
|
Both regressions are fixed in cbdc13d, each reproduced on the previous head before touching anything. Deletion accepts every documented alias. The old guard compared the lookup against the topic heading, so get_topic answered the readable slug and the persisted stem while delete_topic refused both; your probe now deletes through all three and the regression rewrites the topic and deletes by each alias in turn, failing on the previous head at the first slug attempt. The guard itself was doing the write path's job: the lookup resolution already rejects an ambiguous slug and validates digest stems, so the file it names is the record the lookup resolves to, and write_topic retirement keeps the topic ownership check where a resolution can genuinely name another topic's file. Consolidation no longer depends on the file-specific stem. The reload tries the persisted stem first (the exact identity for every file store shape, including explicit slug records whose slug another topic shares), then the topic name and the readable slug, both part of the documented get_topic contract. A conforming custom store whose get_topic resolves by topic or slug answers through the topic alias even though its records carry no stem, so no record is skipped. The regression runs a minimal dict backed MemoryStore through forced consolidation end to end and asserts the record comes back consolidated; on the previous head the count is zero. The three red checks were also mine, all traced and fixed: the package and dependency bounds failures were the now unused pointer trim helper left behind by the index trimming rework, removed here, and the typing check failures were the paused consolidation test's stub, now fully typed with the real signatures. Pyright reports the source file and the test file clean, the memory suite passes at 54 tests, the neighboring harness suites pass at 372, and ruff check and format are clean. |
| because a legacy heading may spell the topic differently than a lookup. | ||
| """ | ||
| record = MemoryTopicRecord.from_markdown(path.read_text(encoding="utf-8"), fallback_topic=path.stem) | ||
| return _normalize_topic(record.topic).casefold() == _normalize_topic(topic).casefold() |
There was a problem hiding this comment.
Manohar Paturi (@ManoharPaturi), writing the slug-shaped topic api after API still resolves the existing API file as an alias, then treats the topics as the same through case-folded ownership comparison and deletes the original file. Please keep case-distinct topics separate and add API/api regressions in both write orders.
| slug=record.slug, | ||
| # The persisted stem, not the readable slug: distinct topics can | ||
| # share a slug, and an index link must point at the real file. | ||
| slug=record.storage_stem, |
There was a problem hiding this comment.
Manohar Paturi (@ManoharPaturi), using the computed storage stem as the public index slug breaks custom MemoryStore implementations: before_run later passes this value to get_topic, while external stores commonly resolve the topic name or readable slug, not a file-store stem. Please keep the public lookup contract stable or use the same topic/slug/stem fallback as reload, with a custom-store before_run regression.
The case folded ownership check let the slug shaped topic api retire the API record through their shared resolution, and the index entry factory put the file store stem in the public slug field that custom stores resolve. Ownership now compares exact normalized topics, the index entry carries the readable slug with the persisted stem in its own link-only field, and injection resolves through the stem, slug, and topic ladder, so conforming custom stores keep working. Regressions cover both case orders and a custom store end to end through before_run. Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
|
Both findings are fixed in 35c8ddf, each reproduced on the previous head first. Case distinct topics stay distinct. The case fold in the ownership check was mine, added for legacy headings spelled differently, and your API after api probe exposed what it really did: the slug shaped lookup resolved the API file as the unambiguous slug holder, the folded comparison called them the same topic, and the write retired the original file outright, not just merged it. The comparison is exact on normalized topics now, which errs toward not retiring, and the regression writes both orders and asserts two files survive with isolated reads; it fails on the previous head with get_topic raising for the first topic written. The index slug contract is stable again. The entry factory puts the readable slug back in the public field, and the persisted stem rides in its own link-only field: pointer lines link the stem when one is present and the slug otherwise, so distinct topics still get distinct destinations while external consumers resolving by slug or topic keep working. Injection resolves selected entries through the same ladder the consolidation reload uses, stem first then slug then topic, so a custom store answers through either documented alias. The regression is a dict backed store whose entries come from the shared factory driven through before_run end to end, asserting the topic content reaches the injected messages; on the previous head the factory filled the slug with the stem, the store could not resolve it, and before_run failed. The memory suite passes at 56 tests, pyright reports the source file clean again, and ruff check and format are clean. |
| else: | ||
| exact_path = topics_dir / f"{legacy_name}.md" | ||
| if exact_path.exists(): | ||
| return exact_path |
There was a problem hiding this comment.
Manohar Paturi (@ManoharPaturi), this exact-file fallback still affects reads, not only destructive cleanup. In an upgraded store where c.md contains C++, get_topic("C#") returns that record. The provider's write_memory path then merges the C# fact into C++ and writes it under the C++ digest stem, so no C# topic is created. The new ownership guard only prevents unlinking the unrelated file; it does not prevent this cross-topic read/merge. Please verify legacy exact-file ownership before returning it, and add a provider-level _merge_memory regression for two topics with the same legacy slug.
| # A resolution through a legacy slug-only file or a shared slug can | ||
| # name another topic's record; retiring it would silently destroy | ||
| # that topic, so only a file holding this very topic is retired. | ||
| previous_path.unlink(missing_ok=True) |
There was a problem hiding this comment.
Manohar Paturi (@ManoharPaturi), this migration is write-then-unlink without recovery. If the unlink fails or the process stops between those operations, both legacy and digest files remain. list_topics() then returns duplicates, delete_topic() removes only the digest file and the stale legacy topic returns, and later writes never retire it because lookup already selects the digest path. Please make the dual-file state reconcile safely and add regressions for coexistence, unlink failure, and delete resurrection.
| entries = store.rebuild_index(session, source_id="src", line_limit=10, line_length=200) | ||
| for entry in entries: | ||
| assert (store._get_topics_directory(session, source_id="src") / f"{entry.storage_stem}.md").exists() | ||
| assert store.get_topic(session, source_id="src", topic=entry.storage_stem).memories == ["kept"] |
There was a problem hiding this comment.
Manohar Paturi (@ManoharPaturi), the current Test Typing Checks (3.11) job fails mypy, pyright, and ty on tests added in this file. Here entry.storage_stem is str | None; the same job reports additional current-head errors at lines 1527, 1534, 1590, 1596, and 1659. Please make the test doubles and assignments satisfy the core typing contracts so the required typing job passes.
|
Both findings reproduced on the previous head first, then fixed together with the typing job. The legacy exact file no longer answers another topic's lookup. Reading C# through the legacy c.md holding C++ returned the C++ record on the previous head, and your provider probe merged the C# fact into C++ with no C# topic ever created; that exact run is now the regression. The exact match splits by shape: a digest bearing stem is an injective identity and answers as the file it names, while a slug only file must hold the lookup's topic (folded for case, because the pre digest scheme stored one file per slug, so a heading spelled with different case is still that one topic). A slug lookup now collects every claiming file and answers only when they all hold one and the same topic, preferring the digest bearing claimant: default slug reloads keep working, and colliding topics stay apart. The migration now reconciles instead of leaving the pair behind. write_topic no longer retires the file the lookup happens to name, because once the digest file exists the lookup never selects the stale copy again; instead it sweeps every other file whose record holds this topic and unlinks those, so a migration whose unlink failed or never ran heals on the next write. delete_topic resolves the record, then removes every file holding that record's topic, so nothing resurrects. list_topics hides a stale copy of a topic that already lives on a digest stem rather than list the topic twice. Three regressions cover the coexistence listing, the simulated unlink failure that heals on the next write, and the delete that leaves no file behind; all three fail on the previous head. The typing job failed on my tests and the failures were real. I reproduced all five checkers locally through the workspace test typing task and fixed the tests against their contracts: the storage stem is narrowed before the lookup call, the paused consolidation task is typed to what _merge_memory actually returns, the custom store's search_transcripts and write_state now match the abstract signatures exactly, and the consolidation client is the same harness client the other tests use instead of a loosely typed duplicate. All five checkers pass on the test tree, the strict source pyright reports clean, the memory suite is at 60 with the four new regressions, and the neighboring harness suites pass at 557. |
Motivation & Context
MemoryFileStorekeyed every topic file by_slugify_topic(topic), which is many to one:C ++andC #both slug toc.write_topicoverwrites the slug path unconditionally, so writing a second topic that shares a slug silently destroyed the first topic's memories. Reproduced on main: after writing both topics,get_topic("C ++")returned the C # memory and the C ++ memory was gone. The store's own path components for owner, session, source, and scope all use the injective_storage_key_segmentderivation for exactly this reason; the topic filename was the one mapping left below that standard.Description & Review Guide
record.slug). When a slug-only lookup is ambiguous because several topics share that slug, the file whose recorded topic matches the lookup wins, else the first stem in sorted order keeps the choice deterministic._existing_topic_path, and the regression test writingC ++thenC #and reading both back, which fails on main.Related Issue
Fixes #9723
AI Assistance
Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.