Skip to content

Python: fix(core): persist compaction annotations in file-backed history - #10359

Open
Yufeng He (he-yufeng) wants to merge 3 commits into
microsoft:mainfrom
he-yufeng:fix/file-history-persist-compaction-annotations
Open

Yufeng He (he-yufeng) wants to merge 3 commits into
microsoft:mainfrom
he-yufeng:fix/file-history-persist-compaction-annotations

Conversation

@he-yufeng

Copy link
Copy Markdown
Contributor

Motivation & Context

FileHistoryProvider.save_messages appends only the messages filter_new_messages reports as unseen. Compaction excludes stored messages by annotating them in place (set_excluded writes additional_properties), so the next save carries the same transcript with changed payloads on records that are already on disk — and writes nothing. The exclusion never reaches the file.

InMemoryHistoryProvider keeps Message objects in session.state, so the identical annotation is durable there. With a file-backed transcript the result is that skip_excluded=True keeps returning the full history and the context grows every turn.

Reproduced on main with a local fake chat client, six turns, SlidingWindowStrategy(keep_last_groups=2):

history provider model-call message counts stored persisted _excluded loaded next turn
InMemoryHistoryProvider 1, 3, 5, 5, 5, 5 12 8 4
FileHistoryProvider 1, 3, 5, 7, 9, 11 12 0 12

Description & Review Guide

  • What are the major changes? When the incoming prefix covers the whole stored transcript, save_messages compares payloads as serialized dicts and rewrites the changed records together with the new tail. Both formats replace the file atomically through a sibling temporary file and os.replace, matching what FileSessionStore already does. MessagePack records are encoded into the staging file, so a record the reader would reject never reaches the history and the previous file stays byte-identical.
  • What is the impact of these changes? Compaction annotations become durable for file-backed history, so skip_excluded behaves the same as with the in-memory provider. The append path is unchanged for the ordinary case: a save with only new messages still appends, and an unchanged replay leaves the file byte-identical with no write and no mtime change. A shorter suffix replay still only appends, so a partial replay never rewrites records it did not carry.
  • What do you want reviewers to focus on? The overlap rule in _updated_stored_messages: it requires len(incoming) - len(new_messages) == len(existing), which is deliberately narrower than "any matched suffix". That keeps the rewrite confined to transcripts the caller actually carried in full, at the cost of not updating annotations when a caller replays only a suffix. Also worth a look: the rewrite holds the same per-file thread and async locks as the append path, so concurrent writers keep the existing serialization.

This does not change CompactionProvider. Its after_run reads session.state[history_source_id]["messages"], which file-backed and vector-backed providers leave empty; making compaction reach those providers is a separate design question and is not attempted here. This change fixes the persistence contract of the provider itself: a save that carries updated stored messages persists them.

Related Issue

N/A - no existing issue covers this. I searched FileHistoryProvider and skip_excluded across issues and PRs; #8546 is about the in-memory provider storing by reference under per-service-call persistence, which is the opposite direction, and no open PR touches FileHistoryProvider. I can open an issue for the provider-side contract if you prefer that ordering.

AI Assistance

  • This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

AI assistance details: implementation, tests, documentation, and this PR description. I reviewed the final diff line by line and ran the suites reported below.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
  • This is not a breaking change. The append path and the reader contract are unchanged; a save that previously wrote nothing now persists the payloads it was given.
Local results on 811ac09ac

Python 3.14.2, agent_framework resolving to this checkout.

  • Full packages/core/tests, non-integration: 9842 tests, 0 failures, 0 errors, 25 skipped (JUnit retained).
  • Red check: with _sessions.py restored to main and the new tests unchanged, 4 failed / 202 passed, failing on KeyError: '_excluded'. Restoring the fix returns the suite to green.
  • Six new parametrized tests over both json and msgpack: annotations reach disk; unchanged replay leaves bytes and mtime untouched; an updated record keeps its position while later messages append; content-identified repeats stay separate records; a rejected batch restores the previous file including the pending annotation; a partial suffix replay appends without rewriting earlier records.
  • Ruff 0.17.4 check and format on both changed Python files: passed. git diff --check: clean.
  • Pyright 1.1.414 over all 107 core source files: 2 errors, both the pre-existing PyYAML compose unknown-type findings in _skills.py. _sessions.py has no diagnostics. This is not a clean full-core typing result.

FileHistoryProvider.save_messages appended only the messages that
filter_new_messages reported as unseen. Compaction excludes stored messages by
annotating them in place, so a save carrying the same transcript with updated
additional_properties wrote nothing and the exclusion was lost on the next read.
InMemoryHistoryProvider keeps Message objects in session state, so the same
annotations were durable there; a file-backed transcript grew without bound
while skip_excluded=True kept returning the full history.

When the incoming prefix covers the whole stored transcript, compare payloads as
serialized dicts and rewrite the changed records along with the new tail. Both
formats go through a sibling temporary file and os.replace, and MessagePack
records are encoded into the staging file so a record the reader would reject
never reaches the history. A shorter suffix replay still only appends, and an
unchanged replay leaves the file byte-identical and untouched.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Transcript rewrites can expose restricted conversation files through permissive staging-file permissions, while long session IDs and type-only property changes also cause annotation updates to fail or be silently skipped.

3 open issues
What changed in this PR

This PR updates Python’s FileHistoryProvider to persist changed payloads, including compaction annotations, when a save replays the full stored transcript. It leaves compaction-provider integration outside this change.

Changes:

  • Adds atomic JSON Lines and MessagePack rewrites while preserving append-only behavior for partial replays.
  • Adds regression tests for annotation persistence, unchanged replays, ordering, and failed rewrites.
  • Documents the full-transcript update contract.
File Description
python/​packages/​core/​tests/​core/​test_sessions.py Adds persistence and replay regression tests for both formats.
python/​packages/​core/​AGENTS.md Documents rewrite behavior and partial-replay limits.
python/​packages/​core/​agent_framework/​_sessions.py Detects changed stored payloads and atomically replaces history files.

Review effort: Balanced

# Encode into a sibling staging file so a rejected record never touches the history.
stage_path = file_path.with_name(f".{file_path.name}.{uuid.uuid4().hex}.tmp")
try:
with stage_path.open("wb") as stage:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reproduced under umask 022: a history chmod'd to 0600 came back 0644 after the rewrite (assert 420 == 384).

Fixed in 326a1a528. Both formats now share _staging_history_file, which reads the history's access bits, creates the staging file with os.open(..., O_WRONLY | O_CREAT | O_EXCL, mode), and re-applies the mode with os.chmod after writing, since os.open masks it with the umask. A history that does not exist yet gets 0600. The probe now reports before 0o600 / after 0o600 under umask 022, and a regression pins it for both formats.

merged = list(existing)
changed = False
for index, message in enumerate(incoming[:stored_count]):
if message.to_dict() != existing[index].to_dict():

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed: a property changed from 1 to True was skipped, and the next load still returned int.

Fixed in 326a1a528 with _same_stored_payload, which requires type(left) is type(right) at every level and recurses through mappings and lists instead of relying on ==. The probe now returns True / bool, and a type-only regression covers both formats.

Comment on lines +2824 to +2838
stage_path = file_path.with_name(f".{file_path.name}.{uuid.uuid4().hex}.tmp")
try:
with stage_path.open("wb") as stage:
for message in [*merged, *new_messages]:
self._encode_msgpack_record(message, record)
stage.write(len(record).to_bytes(self._MSGPACK_RECORD_HEADER_BYTES, "big"))
stage.write(record)
os.replace(stage_path, file_path)
finally:
stage_path.unlink(missing_ok=True)

@staticmethod
def _replace_history_file(file_path: Path, serialized: bytes) -> None:
"""Write ``serialized`` through a sibling temporary file and replace the history."""
temp_path = file_path.with_name(f".{file_path.name}.{uuid.uuid4().hex}.tmp")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed on APFS: a 240-character literal-safe session ID produces a 246-byte .jsonl / 248-byte .msgpack name that the append path accepts, and the rewrite failed with OSError errno=63.

Fixed in 326a1a528: the staging name is now .history-<uuid4 hex>.tmp, independent of the history filename. Both formats rewrite that session successfully, and the regression also asserts no .history-*.tmp is left behind.

…g IDs

All three findings reproduced locally before changing anything.

Restricted transcripts: under umask 022 a 0600 history was replaced by a 0644
staging file. Both rewrites now go through one staging helper that creates the
file with os.open(..., O_EXCL, mode) using the history's own access bits (0600
when it does not exist yet) and re-applies them after writing, since os.open
masks the mode with the umask.

Type-only changes: dict equality treats 1 and True as equal, so a payload that
changed only in type was skipped. Comparison now walks the serialized payload
and requires matching types at every level.

Long session IDs: a 240-character literal-safe ID produces a 246/248-byte
history filename that the append path accepts, but the staging name embedded
that filename and failed with ENAMETOOLONG. The staging name is now a short
fixed prefix plus the random token.

Adds six regressions (both formats): permissions preserved under umask 022,
type-only update persisted, and a long-ID rewrite that also asserts no staging
file is left behind.
@he-yufeng

Copy link
Copy Markdown
Contributor Author

All three reproduced locally on 811ac09ac before changing anything, then fixed in 326a1a528.

Restricted transcripts. With umask 022, a history chmod'd to 0600 came back 0644 after the rewrite (assert 420 == 384). Both formats now share one staging helper: it reads the history's access bits, creates the staging file with os.open(..., O_WRONLY | O_CREAT | O_EXCL, mode), and re-applies the mode with os.chmod after writing because os.open masks it with the umask. A history that does not exist yet gets 0600. Probe after the fix: before 0o600 / after 0o600 under umask 022.

Type-only payload changes. Confirmed: a property changed from 1 to True was skipped and the next load still returned int. Comparison now walks the serialized payload and requires type(left) is type(right) at every level, recursing through mappings and lists. Probe after the fix returns True / bool.

Staging filename length. Confirmed on APFS: a 240-character literal-safe session ID yields a 246-byte .jsonl / 248-byte .msgpack name that the append path accepts, and the rewrite failed with OSError errno=63. The staging name is now .history-<uuid4 hex>.tmp, independent of the history filename. Both formats now rewrite that session successfully, and the test also asserts no .history-*.tmp is left behind.

Six new regressions, both formats. Red check with the previous _sessions.py and these tests unchanged: 6 failed, with exactly the three symptoms above (assert 420 == 384, assert 1 is True, Errno 63 File name too long). After the fix, full packages/core/tests non-integration: 9848 tests, 0 failures, 0 errors, 25 skipped.

I also checked the failure path explicitly for both formats: forcing an encode rejection leaves the history byte-identical and removes the staging file. Ruff 0.17.4 check and format pass; Pyright 1.1.414 over 107 core files reports only the two pre-existing _skills.py PyYAML findings.

@surlymochan Nick (surlymochan) left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the core of this change and the surrounding machinery rather than just the new tests:

  • The reused-buffer concern in the msgpack rewrite is safe: msgspec's Encoder.encode_into overwrites the buffer from position 0 (verified locally — encoding {'a': 1} then {'b': 2} into the same bytearray leaves only the second record, length 4), so _rewrite_msgpack_messages writing len(record) + record per message is correct, matching the existing append path.
  • _updated_stored_messages correctly requires the incoming prefix to cover the whole stored transcript (stored_count == len(existing)), so the partial-replay test's "no rewrite of records it did not carry" holds.
  • The rollback and permission tests cover the failure modes that matter (rejected record leaves the old file byte-identical; a 0600 transcript stays 0600 after a rewrite).

Two questions:

  1. The rewrite decision compares message.to_dict() of the incoming messages against the stored ones. Is to_dict() stable across reconstructions of the same run — i.e. does it carry any volatile field (generated id, timestamp) that would differ on a fresh reconstruction and force a full rewrite on every save even when nothing meaningful changed? The "skips rewrite when nothing changed" test replays the same objects, so it would not catch a volatile field.
  2. The JSON rewrite materializes the whole merged transcript into one string before writing, while the msgpack rewrite streams record by record. For very large transcripts that is a full in-memory copy plus the staging file; is that accepted (the JSON read path already loads the whole file), or worth streaming line by line like the append path does?

Non-blocking: the change is well tested across both formats, including the type-only (1 vs True) payload case.

return len(left_items) == len(right_items) and all(
_same_stored_payload(a, b) for a, b in zip(left_items, right_items, strict=True)
)
return bool(left == right)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nan never equals itself, so once any stored message carries a NaN in additional_properties (a score, a usage metric) every replay counts it as changed and rewrites the whole file on each save. checked both json and msgpack, the inode changes on an identical replay

Suggested change
return bool(left == right)
if isinstance(left, float) and math.isnan(left):
return math.isnan(right)
return bool(left == right)

fixed this plus a test in this commit, you can just cherry pick it instead of redoing it: its-amann@829fa56

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Cherry-picked as ce986772184cd97e53b23b5ac416d540233d41f7, preserving the original commit and authorship.

Validation on that head:

  • The new JSON and MessagePack cases both pass: 2 passed, 212 deselected.
  • Removing the two-line NaN guard makes both cases fail with the inode changing on identical replay (pytest exit 1). I restored the production file byte for byte, verified its SHA256, and reran the cases successfully.
  • Full test_sessions.py: 214 passed.
  • Ruff check and format check pass on both changed files.
  • Pyright on _sessions.py: 0 errors.

The tests used Python 3.14.5, pytest 9.2.0, msgspec 0.22.0 and pydantic 2.14.1 in an existing non-iCloud environment, with agent_framework imported from this branch.

This branch was successfully deployed

1 active deployment
github-app-auth — ce986772 Deployed Nov 16, 2026 by he-yufeng via add_label #27905
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Usage: [Issues, PRs], Target: documentation in the code base and learn docs python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants