Performance: full workspace chat cache rewrites cause CPU spikes as history grows

[itkonen]

ECA persists all chats for a workspace in one db.transit.json. Most persisted history additions invoke update-workspaces-cache!, which normalizes the complete :chats map, Transit-encodes it into a temporary file, and renames that file over the previous cache.

As workspace history grows, the cost of adding one assistant message, reasoning result, or tool result therefore scales with all retained workspace history, including unrelated chats and completed subagent chats.

This appears to cause:

• brief 100% CPU spikes at chat-step boundaries;
• substantial spikes after tool results;
• increasing latency as a workspace accumulates history;
• repeated serialization and disk output during tool-heavy turns;
• expensive startup deserialization.

Relevant code

• add-to-history! persists most appended history records
• update-workspaces-cache! selects and normalizes the complete :chats map
• upsert-cache! serializes the complete value to a temporary file and replaces the destination
• Each completed tool normally appends tool_call_output, triggering a save
• Rejected tools use the same call/output pattern
• finish-chat-prompt! performs another workspace-cache save
• Error handling includes additional defensive saves
• Initialization reads and decodes the complete cache

The durable save cadence was introduced in “Persist chats more durably so /resume keeps long and errored chats” on May 22, 2026. The durability goal seems valid; this issue is about the resulting performance characteristics once the cache becomes large.

To Reproduce

1. Use ECA normally in one workspace with persistent chat caching enabled.
2. Accumulate many chats or subagent runs, especially tool-heavy runs.
3. Observe the size of the workspace’s db.transit.json.
4. Start a prompt that performs one or more tool calls.
5. Observe CPU usage and the cache file’s modification time while tool results are returned.
6. Restart ECA and observe initialization while the complete cache is decoded.

In the workspace where I observed the problem, a sanitized snapshot contained:

Measurement	Value
Cache size	315,964,557 bytes / 301.33 MiB
Chats	726
Subagent chats	700
Messages	76,842
Tool calls	28,066
Tool outputs	28,066
Reasoning messages	18,615

No chat IDs, workspace IDs, titles, prompts, paths, tool arguments, tool outputs, or other message content are included here.

Repeated full-cache measurements on this machine produced:

Operation	Observed range
Transit decode	1.17–1.42 seconds
Transit encode to a null output stream	0.98–1.15 seconds
Maximum benchmark-process RSS	1.73 GiB

The encode measurement excludes writing the approximately 301 MiB temporary file and renaming it.

At the measured cache size:

Physical save requests	Logical payload serialized/written
1	approximately 301 MiB
10	approximately 2.94 GiB
100	approximately 29.43 GiB

These are application-level payload estimates. Physical device writes will vary with filesystem caching and other operating-system behavior.

A typical tool-heavy turn can perform:

1. A save for assistant text before the tool batch, if present.
2. One save for each completed or rejected tool result.
3. A save for final assistant text.
4. Another save from prompt completion.

This is not a save per streamed token. The issue occurs when completed history records are appended and at lifecycle persistence boundaries.

Expected behavior

The cost of persisting one chat-history mutation should remain reasonably bounded as unrelated chats and completed subagent histories accumulate.

Durable mid-turn recovery should not require repeatedly serializing and replacing hundreds of megabytes of unrelated history for each tool result or lifecycle transition.

Additional context

Impact

The observed behavior has several related effects:

• A single small history mutation can trigger serialization of the complete workspace history.
• Parallel tool results can queue several complete saves.
• Completed subagent histories contribute to every later save, even when unrelated to the active chat.
• Normal completion may save both when the assistant message is appended and again during prompt finalization.
• Defensive error saves can repeat the same expensive operation during one logical failure transition.
• Startup eagerly reads and materializes the complete cache.

The temporary-file-and-rename implementation protects against partially written destination files, but it does not reduce the amount of data serialized.

Cache composition

A content-size proxy for the sanitized snapshot indicated that the largest contributors were:

Role	Approximate content-size proxy
tool_call_output	185.97 MiB
tool_call	45.79 MiB
reason	39.68 MiB
assistant	5.74 MiB
user	2.11 MiB

These are UTF-8 byte counts of printed message content, not exact Transit-file byte attribution. They are included only to show that tool-heavy and subagent-heavy histories can grow the workspace cache quickly.

Retention

Startup cleanup currently uses a default retention period of 14 days:

• Cleanup scheduling
• cleanup-old-chats!

The cleanup comparison uses :created-at, rather than recent activity through :updated-at. Cleanup also performs another complete cache rewrite when chats are removed.

This may be intentional and is secondary to the write-amplification issue, but it may affect how quickly active workspaces accumulate large caches.

Possible directions for discussion

I do not have a specific fix to propose because the right approach depends on ECA’s desired durability and history-retention semantics.

Possible design areas include:

• reducing redundant saves at lifecycle boundaries;
• coalescing overlapping save requests;
• persisting only the changed chat;
• using incremental or append-oriented persistence;
• separating chat metadata from message history;
• changing completed-subagent retention;
• limiting or externalizing persisted tool-output history;
• warning or exposing metrics when a workspace cache becomes unusually large.

Coalescing alone would reduce repeated writes in a burst, but one save would still serialize the entire cache. More incremental storage would be a larger architectural change.

Secondary CPU observations

While investigating, I found two additional whole-history/request processing paths. These are separate from cache-write amplification and may deserve separate issues.

Request logging formats the request before checking whether debug logging is enabled:

• llm-util/log-request
• logger/debug checks the level after its arguments have been evaluated

This can stringify a complete provider request even when the resulting debug message is discarded.

Context-usage breakdown traverses and stringifies the complete message history:

• Usage callback
• context-breakdown

These may contribute to smaller request/usage-boundary CPU spikes, but they do not explain the repeated 301 MiB cache writes.

Open Questions

1. Is a durable complete-cache save intended after every persisted history append, including every subagent tool result?
2. Is a workspace cache of hundreds of megabytes considered within the expected operating range?
3. How much history is expected to survive a crash during a long tool or subagent loop?
4. Is detailed completed-subagent history intended to remain indefinitely available until generic chat retention removes it?
5. Is retention intentionally based on :created-at rather than :updated-at?
6. Would it be useful for :db/upsert-cache metrics to expose serialized bytes and save duration, or warn above a size/latency threshold?
7. Should the request-log formatting and context-breakdown observations be filed separately?

[ovistoica]

I was going to open the same issue as I had the same problem and had to sadly revert to retention of 14 days as each message append to chat history was taking over 1 second given we store the entirety of sessions in 1 DB.

I propose a design very similar to with pi is doing (currently I think the most performant one):

• 1 append only file per chat (not per directory / project)
• JSONL instead of transit json for fastest append operations

As an added bonus, I think this type of storage would even permit us to create branching in the same chat (different from forking) so you can have even more control over the chat history

CC @ericdallo @zikajk

[zikajk]

Makes sense, but it's not a small refactor. Also, it raises the question if an embedded DB (SQLite / Datalevin) wouldn't be better than JSONL as it would make some operations simpler.

[ovistoica]

@zikajk For sure! I haven't considered that but it is a good idea. If I were to choose, I think Datalevin would be better since it more easily captures graph data, that can be used to provide futher conversation branching/resuming (I think)

[zikajk]

@ovistoica We'll see what @ericdallo thinks about it. Unfortunately, Datalevin lacks Apple Intel support, and apparently we have users like that. On the other hand, files have the advantage of being more accessible. Tough decision :-)