Repository navigation
Python: fix(core): persist compaction annotations in file-backed history - #10359
Yufeng He (he-yufeng) wants to merge 3 commits into
Conversation
FileHistoryProvider.save_messages appended only the messages that filter_new_messages reported as unseen. Compaction excludes stored messages by annotating them in place, so a save carrying the same transcript with updated additional_properties wrote nothing and the exclusion was lost on the next read. InMemoryHistoryProvider keeps Message objects in session state, so the same annotations were durable there; a file-backed transcript grew without bound while skip_excluded=True kept returning the full history. When the incoming prefix covers the whole stored transcript, compare payloads as serialized dicts and rewrite the changed records along with the new tail. Both formats go through a sibling temporary file and os.replace, and MessagePack records are encoded into the staging file so a record the reader would reject never reaches the history. A shorter suffix replay still only appends, and an unchanged replay leaves the file byte-identical and untouched.
There was a problem hiding this comment.
🟡 Changes recommended
Transcript rewrites can expose restricted conversation files through permissive staging-file permissions, while long session IDs and type-only property changes also cause annotation updates to fail or be silently skipped.
3 open issues
What changed in this PR
This PR updates Python’s FileHistoryProvider to persist changed payloads, including compaction annotations, when a save replays the full stored transcript. It leaves compaction-provider integration outside this change.
Changes:
- Adds atomic JSON Lines and MessagePack rewrites while preserving append-only behavior for partial replays.
- Adds regression tests for annotation persistence, unchanged replays, ordering, and failed rewrites.
- Documents the full-transcript update contract.
| File | Description |
|---|---|
python/packages/core/tests/core/test_sessions.py |
Adds persistence and replay regression tests for both formats. |
python/packages/core/AGENTS.md |
Documents rewrite behavior and partial-replay limits. |
python/packages/core/agent_framework/_sessions.py |
Detects changed stored payloads and atomically replaces history files. |
Review effort: Balanced
| # Encode into a sibling staging file so a rejected record never touches the history. | ||
| stage_path = file_path.with_name(f".{file_path.name}.{uuid.uuid4().hex}.tmp") | ||
| try: | ||
| with stage_path.open("wb") as stage: |
There was a problem hiding this comment.
Reproduced under umask 022: a history chmod'd to 0600 came back 0644 after the rewrite (assert 420 == 384).
Fixed in 326a1a528. Both formats now share _staging_history_file, which reads the history's access bits, creates the staging file with os.open(..., O_WRONLY | O_CREAT | O_EXCL, mode), and re-applies the mode with os.chmod after writing, since os.open masks it with the umask. A history that does not exist yet gets 0600. The probe now reports before 0o600 / after 0o600 under umask 022, and a regression pins it for both formats.
| merged = list(existing) | ||
| changed = False | ||
| for index, message in enumerate(incoming[:stored_count]): | ||
| if message.to_dict() != existing[index].to_dict(): |
There was a problem hiding this comment.
Confirmed: a property changed from 1 to True was skipped, and the next load still returned int.
Fixed in 326a1a528 with _same_stored_payload, which requires type(left) is type(right) at every level and recurses through mappings and lists instead of relying on ==. The probe now returns True / bool, and a type-only regression covers both formats.
| stage_path = file_path.with_name(f".{file_path.name}.{uuid.uuid4().hex}.tmp") | ||
| try: | ||
| with stage_path.open("wb") as stage: | ||
| for message in [*merged, *new_messages]: | ||
| self._encode_msgpack_record(message, record) | ||
| stage.write(len(record).to_bytes(self._MSGPACK_RECORD_HEADER_BYTES, "big")) | ||
| stage.write(record) | ||
| os.replace(stage_path, file_path) | ||
| finally: | ||
| stage_path.unlink(missing_ok=True) | ||
|
|
||
| @staticmethod | ||
| def _replace_history_file(file_path: Path, serialized: bytes) -> None: | ||
| """Write ``serialized`` through a sibling temporary file and replace the history.""" | ||
| temp_path = file_path.with_name(f".{file_path.name}.{uuid.uuid4().hex}.tmp") |
There was a problem hiding this comment.
Confirmed on APFS: a 240-character literal-safe session ID produces a 246-byte .jsonl / 248-byte .msgpack name that the append path accepts, and the rewrite failed with OSError errno=63.
Fixed in 326a1a528: the staging name is now .history-<uuid4 hex>.tmp, independent of the history filename. Both formats rewrite that session successfully, and the regression also asserts no .history-*.tmp is left behind.
…g IDs All three findings reproduced locally before changing anything. Restricted transcripts: under umask 022 a 0600 history was replaced by a 0644 staging file. Both rewrites now go through one staging helper that creates the file with os.open(..., O_EXCL, mode) using the history's own access bits (0600 when it does not exist yet) and re-applies them after writing, since os.open masks the mode with the umask. Type-only changes: dict equality treats 1 and True as equal, so a payload that changed only in type was skipped. Comparison now walks the serialized payload and requires matching types at every level. Long session IDs: a 240-character literal-safe ID produces a 246/248-byte history filename that the append path accepts, but the staging name embedded that filename and failed with ENAMETOOLONG. The staging name is now a short fixed prefix plus the random token. Adds six regressions (both formats): permissions preserved under umask 022, type-only update persisted, and a long-ID rewrite that also asserts no staging file is left behind.
|
All three reproduced locally on Restricted transcripts. With umask 022, a history chmod'd to 0600 came back 0644 after the rewrite ( Type-only payload changes. Confirmed: a property changed from Staging filename length. Confirmed on APFS: a 240-character literal-safe session ID yields a 246-byte Six new regressions, both formats. Red check with the previous I also checked the failure path explicitly for both formats: forcing an encode rejection leaves the history byte-identical and removes the staging file. Ruff 0.17.4 check and format pass; Pyright 1.1.414 over 107 core files reports only the two pre-existing |
Nick (surlymochan)
left a comment
There was a problem hiding this comment.
Verified the core of this change and the surrounding machinery rather than just the new tests:
- The reused-buffer concern in the msgpack rewrite is safe:
msgspec'sEncoder.encode_intooverwrites the buffer from position 0 (verified locally — encoding{'a': 1}then{'b': 2}into the samebytearrayleaves only the second record, length 4), so_rewrite_msgpack_messageswritinglen(record)+recordper message is correct, matching the existing append path. _updated_stored_messagescorrectly requires the incoming prefix to cover the whole stored transcript (stored_count == len(existing)), so the partial-replay test's "no rewrite of records it did not carry" holds.- The rollback and permission tests cover the failure modes that matter (rejected record leaves the old file byte-identical; a 0600 transcript stays 0600 after a rewrite).
Two questions:
- The rewrite decision compares
message.to_dict()of the incoming messages against the stored ones. Isto_dict()stable across reconstructions of the same run — i.e. does it carry any volatile field (generated id, timestamp) that would differ on a fresh reconstruction and force a full rewrite on every save even when nothing meaningful changed? The "skips rewrite when nothing changed" test replays the same objects, so it would not catch a volatile field. - The JSON rewrite materializes the whole merged transcript into one string before writing, while the msgpack rewrite streams record by record. For very large transcripts that is a full in-memory copy plus the staging file; is that accepted (the JSON read path already loads the whole file), or worth streaming line by line like the append path does?
Non-blocking: the change is well tested across both formats, including the type-only (1 vs True) payload case.
| return len(left_items) == len(right_items) and all( | ||
| _same_stored_payload(a, b) for a, b in zip(left_items, right_items, strict=True) | ||
| ) | ||
| return bool(left == right) |
There was a problem hiding this comment.
nan never equals itself, so once any stored message carries a NaN in additional_properties (a score, a usage metric) every replay counts it as changed and rewrites the whole file on each save. checked both json and msgpack, the inode changes on an identical replay
| return bool(left == right) | |
| if isinstance(left, float) and math.isnan(left): | |
| return math.isnan(right) | |
| return bool(left == right) |
fixed this plus a test in this commit, you can just cherry pick it instead of redoing it: its-amann@829fa56
There was a problem hiding this comment.
Thanks. Cherry-picked as ce986772184cd97e53b23b5ac416d540233d41f7, preserving the original commit and authorship.
Validation on that head:
- The new JSON and MessagePack cases both pass: 2 passed, 212 deselected.
- Removing the two-line NaN guard makes both cases fail with the inode changing on identical replay (pytest exit 1). I restored the production file byte for byte, verified its SHA256, and reran the cases successfully.
- Full
test_sessions.py: 214 passed. - Ruff check and format check pass on both changed files.
- Pyright on
_sessions.py: 0 errors.
The tests used Python 3.14.5, pytest 9.2.0, msgspec 0.22.0 and pydantic 2.14.1 in an existing non-iCloud environment, with agent_framework imported from this branch.


Motivation & Context
FileHistoryProvider.save_messagesappends only the messagesfilter_new_messagesreports as unseen. Compaction excludes stored messages by annotating them in place (set_excludedwritesadditional_properties), so the next save carries the same transcript with changed payloads on records that are already on disk — and writes nothing. The exclusion never reaches the file.InMemoryHistoryProviderkeepsMessageobjects insession.state, so the identical annotation is durable there. With a file-backed transcript the result is thatskip_excluded=Truekeeps returning the full history and the context grows every turn.Reproduced on
mainwith a local fake chat client, six turns,SlidingWindowStrategy(keep_last_groups=2):_excludedInMemoryHistoryProviderFileHistoryProviderDescription & Review Guide
save_messagescompares payloads as serialized dicts and rewrites the changed records together with the new tail. Both formats replace the file atomically through a sibling temporary file andos.replace, matching whatFileSessionStorealready does. MessagePack records are encoded into the staging file, so a record the reader would reject never reaches the history and the previous file stays byte-identical.skip_excludedbehaves the same as with the in-memory provider. The append path is unchanged for the ordinary case: a save with only new messages still appends, and an unchanged replay leaves the file byte-identical with no write and no mtime change. A shorter suffix replay still only appends, so a partial replay never rewrites records it did not carry._updated_stored_messages: it requireslen(incoming) - len(new_messages) == len(existing), which is deliberately narrower than "any matched suffix". That keeps the rewrite confined to transcripts the caller actually carried in full, at the cost of not updating annotations when a caller replays only a suffix. Also worth a look: the rewrite holds the same per-file thread and async locks as the append path, so concurrent writers keep the existing serialization.This does not change
CompactionProvider. Itsafter_runreadssession.state[history_source_id]["messages"], which file-backed and vector-backed providers leave empty; making compaction reach those providers is a separate design question and is not attempted here. This change fixes the persistence contract of the provider itself: a save that carries updated stored messages persists them.Related Issue
N/A - no existing issue covers this. I searched
FileHistoryProviderandskip_excludedacross issues and PRs; #8546 is about the in-memory provider storing by reference under per-service-call persistence, which is the opposite direction, and no open PR touchesFileHistoryProvider. I can open an issue for the provider-side contract if you prefer that ordering.AI Assistance
AI assistance details: implementation, tests, documentation, and this PR description. I reviewed the final diff line by line and ran the suites reported below.
Contribution Checklist
Local results on
811ac09acPython 3.14.2,
agent_frameworkresolving to this checkout.packages/core/tests, non-integration: 9842 tests, 0 failures, 0 errors, 25 skipped (JUnit retained)._sessions.pyrestored tomainand the new tests unchanged, 4 failed / 202 passed, failing onKeyError: '_excluded'. Restoring the fix returns the suite to green.jsonandmsgpack: annotations reach disk; unchanged replay leaves bytes and mtime untouched; an updated record keeps its position while later messages append; content-identified repeats stay separate records; a rejected batch restores the previous file including the pending annotation; a partial suffix replay appends without rewriting earlier records.git diff --check: clean.composeunknown-type findings in_skills.py._sessions.pyhas no diagnostics. This is not a clean full-core typing result.