fix(core): compact repeated message diff events - #48537
Conversation
Avoid re-appending a summarized user message's full patch array in each message.updated.1 event. Preserve the first durable diff event and projection value for replay compatibility while compacting later updates.\n\nThe focused SQLite regression fixture reduces the repeated event row from 262515 bytes to 284 bytes (99.89%). Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Record the chosen replay-compatible event compaction, failing-before and passing-after output, SQLite byte measurement, verification, and residual risk. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Compact only patch bodies when SQLite-normalized summary diffs exactly match the projected value. Changed summaries retain their full new patches, and replay restores compact event patches without decoding every projection.\n\nThe real summarize regression covers changed diffs and fresh replay. The repeated fixture event falls from 262515 bytes to 360 bytes (99.86%). Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Keep the requested evidence report as a local untracked deliverable rather than adding it to the upstream branch. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
|
Thanks for your contribution! This PR doesn't have a linked issue. All PRs must reference an existing issue. Please:
See CONTRIBUTING.md for details. |
|
This PR doesn't fully meet our contributing guidelines and PR template. What needs to be fixed:
Please edit this PR description to address the above within 2 hours, or it will be automatically closed. If you believe this was flagged incorrectly, please let a maintainer know. |
|
The following comment was made by an LLM, it may be inaccurate: Found a related PR: #42771 - This PR appears to address the same core issue: removing patch text from event payloads to reduce database bloat. It likely represents a previous approach or related work on the same problem of unbounded growth from repeated message diff events. |
|
This pull request has been automatically closed because it was not updated to meet our contributing guidelines within the 2-hour window. Feel free to open a new pull request that follows our guidelines. |
Problem
SessionSummary.summarizeattaches the full git patch text of a turn to the user message atsummary.diffs(packages/opencode/src/session/summary.ts). Every latersessions.updateMessageon that same user message re-publishesMessageUpdatedwith the entiresummarypayload, and because the event log is immutable each publish writes a brand new row containing another full copy of every patch.The result is unbounded growth: a single turn that touches a large file is re-serialized into the event log on every subsequent update of that message.
Measured impact
On a real
opencode.dbthat had grown to 79 GB, sampling 400message.updated.1rows:datasizesummaryThat extrapolates to roughly 49 GB of the 79 GB database being duplicated patch text.
Fix
The patch text only needs to reach the projector once. When a user message is updated and its outgoing
summary.diffsare byte-identical to what is already stored, the published event now carries the diff metadata (file,additions,deletions,status) with the heavypatchstring omitted.packages/opencode/src/session/session.tscompares the outgoing diffs against the stored row inside SQLite (json_extract(...) = json(?)) and stripspatchonly on an exact match.packages/core/src/session/projector.tsrestores the stored patches when, and only when, the incoming event is exactly that stripped form: same length, every item missingpatch, and all metadata equal to the stored item.Fails safe by construction
Any mismatch at all, including a mere JSON key ordering difference, makes the comparison fail and the full payload is published unchanged. The optimization is lost in that case; the data never is.
Backward compatible
Events already persisted carry real patch strings, so the restore guard never triggers for them and they project exactly as before. No schema change, no migration, no change to the public
MessageUpdatedpayload type.Cost
The projector performs one extra row read, and only for events that actually arrive in the compacted form.
Verification
test/server/session-diff-missing-patch.test.tsgains two cases alongside the two existing ones:keeps stored turn diffs while compacting later message update eventsproves the compaction round-trips.persists changed turn diffs through a fresh event replayproves that when a turn's diffs genuinely change, the new value is published, persisted and survives replay. This pins the failure mode an earlier revision of this change introduced, where diffs were skipped whenever any value was already stored rather than only when it was identical.In a SQLite fixture reproducing the repeated-update pattern, the repeated event rows drop from 262,515 bytes to 284 bytes while the stored turn diff stays fully available through
GET /session/<id>/diff.