fix(history): bound external CLI replay and turn-index rebuild#504
fix(history): bound external CLI replay and turn-index rebuild#504ShiboSheng wants to merge 31 commits into
Conversation
Large SQLite-backed sessions previously collected every event row and copied full argument, result, and content JSON before determining whether the row could contribute turn metadata. That made index migration memory grow with the complete transcript. Fold rusqlite rows directly into turn accumulators, classify tool payload requirements before copying large fields, and publish rebuilt summaries transactionally. Unknown tools retain the conservative metadata behavior so the optimization does not change native SDE results. Verification: - `cargo test -p session_persistence turn_index`: PASS - `cargo test -p orgtrack_core`: PASS Pre-commit hook ran. Total eslint: 0, total circular: 1
The storage-specific replay drivers require incremental SQLite BLOB access, native file notifications, compressed streaming support, stable hashing, and process-level RSS assertions that were not enabled in the existing workspace dependency set. Enable the required rusqlite BLOB feature and add the notify, flate2, sha2, and libc wiring used by replay indexing, streamed export, and performance tests. Keep the lockfile aligned with the workspace manifests. Verification: - `cargo check --workspace --all-targets`: PASS on the final branch - Cargo lockfile resolution completed with hooks enabled Pre-commit hook ran. Total eslint: 0, total circular: 1
External history sources do not share one storage format: some append JSONL, some update SQLite or WAL rows, Cursor and Warp use specialized records, and Cline rewrites a whole JSON document. Treating them as one full-history loader caused repeated parsing and session-sized memory growth. Add the authoritative 15-source registry, storage-specific JSONL, SQLite, structured, and whole-document drivers, compact replay index tables, stable cursors, bounded windows, payload locators, and source conformance fixtures. Each source keeps its own parsing and tool-pairing semantics while sharing the same public replay contract. Verification: - `cargo test -p orgtrack_core`: 453 passed, 3 ignored - 30 MiB and 300 MiB replay performance fixtures: PASS - Real 335 MiB Codex Issue 272 core replay: PASS Pre-commit hook ran. Total eslint: 0, total circular: 1
A compact source index alone did not remove the memory amplification because application commands, managed CLI mirrors, EventStore hydration, metadata, provenance, and export paths could still request or materialize complete histories. Add the Rust replay commands, watcher lifecycle, bounded EventStore application, managed transcript routing, streamed export, collaboration snapshot ingestion, and compact metadata consumers. Keep replay state separate from native SDE persistence and reject mismatched source or session identities. Verification: - `cargo test -p org2 agent_sessions::event_pipeline`: 403 passed - 327 MiB streamed export hash and buffer-bound test: PASS - Native SDE replay-command isolation assertions: PASS Pre-commit hook ran. Total eslint: 0, total circular: 1
Even with bounded Rust readers, renderer and background consumers could preserve the old full-transcript behavior by loading all chunks, sending events back to Rust, or building complete arrays for pagination, Fork, Cloud, Raw Transcript, and collaboration flows. Introduce the typed replay RPC and SessionCore transport, route imported and managed CLI sessions through bounded windows and deltas, virtualize transcript payload access, and migrate Fork, Cloud, hover, provenance, turn loading, and session switching. Native SDE adapters continue using their existing persisted-database mode. Verification: - Full Vitest suite on the final branch: 6,197 passed - `pnpm typecheck`: PASS - Replay RPC schema, request guard, pagination, lifecycle, Fork, Cloud, and transcript tests: PASS Pre-commit hook ran. Total eslint: 0, total circular: 1
Bring the issue branch up to the develop snapshot ending at build v1.2.1 so the replay implementation is evaluated against the current source parsers, session directory, cloud synchronization, and dispatch behavior. Resolve source-adapter and cloud-consumer conflicts by retaining bounded replay ownership of external history while incorporating the develop-side parsing and lifecycle changes. No protected local artifacts are included. Verification: - Subsequent branch-wide Rust and Vitest gates recorded in the Issue 443 audit: PASS - Final branch tree is checked independently after the message-only rewrite Pre-commit hook ran. Total eslint: 0, total circular: 1
Cline can expose a temporarily invalid whole-document rewrite and Cursor CLI can temporarily present an incompatible lineage. Re-evaluating the same rejected physical snapshot on every poll wasted CPU and repeatedly exercised large parsing paths. Persist a compact rejection watermark keyed by source identity, parser version, size, timestamp, and fingerprint. Continue serving the previous valid generation until the physical source changes, then clear the watermark only after a valid generation is published atomically. Verification: - Rejected Cline and Cursor CLI snapshot regression tests: PASS - `cargo test -p orgtrack_core`: PASS Pre-commit hook ran. Total eslint: 0, total circular: 1
The first prewarm implementation returned a bounded SessionEvent array to JavaScript and then sent the same array back to Rust for EventStore application. That preserved an avoidable Rust to JS to Rust copy and allowed stale prewarm completions to outlive a session visit. Read, normalize, persist Shell payloads, and apply the bounded window directly in Rust. Add prewarm episode guards, reject native SDE identities before allocating replay state, and keep foreground watchers independent from read-only prewarming. Verification: - Replay RPC schema and Cursor prewarm guard tests: PASS - `cargo test -p org2 agent_sessions::event_pipeline`: PASS - Native SDE external-replay call count remains zero Pre-commit hook ran. Total eslint: 0, total circular: 1
Several secondary paths could still retain session-sized data after the initial adapter work: replay caches lacked a complete byte policy, payload roots could be decoded eagerly, and Shell, CLI, exporter, and provenance consumers could bypass bounded projection. Add byte-aware LRU and TTL cache policy, explicit payload encodings and bounded body projections, canonical Shell artifacts, serialized cache access, source routing contracts, and bounded loaders for CLI, export, and provenance. Preserve exact range hydration for callers that explicitly request the original payload. Verification: - 30 MiB to 300 MiB bounded-growth tests: PASS - 10 MiB Shell subprocess and RSS plateau stress tests: PASS - `cargo test -p agent_core`: 3,090 passed, 2 ignored - `cargo test -p perf_utils`: 68 passed Pre-commit hook ran. Total eslint: 0, total circular: 1
Small unit fixtures did not prove the hard memory, IPC, cursor, reset, lifecycle, and source-registration requirements behind Issue 443, and they did not protect the native SDE and prior memory fixes from regression. Add deterministic large-history performance cases, cache and schema assertions, rendered replay coverage, and independent architecture and frontend audit reports. Cover hard event limits, source parity, unchanged polling, streamed payloads, and the Issue 425, 435, and 446 compatibility boundaries. Verification: - Bounded replay performance and cache tests: PASS - Replay RPC contract tests: PASS - Rendered chat replay scenarios: PASS Pre-commit hook ran. Total eslint: 0, total circular: 1
Update the issue branch to the develop snapshot that makes release downloads use bounded retries, avoiding a late integration against stale updater and application scaffolding. The merge required no replay-specific behavior change. The bounded external-history implementation and native SDE boundary remain unchanged. Verification: - Subsequent branch-wide Rust and Vitest gates recorded in the Issue 443 audit: PASS - Final branch tree is checked independently after the message-only rewrite Pre-commit hook ran. Total eslint: 0, total circular: 1
Concurrent opens and secondary readers could update the same compact replay cache at the same time, producing avoidable duplicate work and risking stale state publication across OpenCode, export, history, and provenance consumers. Serialize cache mutation per source session, keep read-only consumers on bounded query paths, and prevent stale renderer retention from reintroducing a released replay window. Preserve generation and revision guards while avoiding duplicate EventStore work. Verification: - `cargo test -p orgtrack_core`: PASS - `cargo test -p org2 agent_sessions::event_pipeline`: PASS - Session-scoped retention and replay schema tests: PASS Pre-commit hook ran. Total eslint: 6, total circular: 0
Synthetic fixtures could not reproduce the real cost of opening, switching away from, and reopening the 335 MiB Codex Issue 272 session through the rendered Tauri application. Add an isolated real-session E2E path that discovers the source through the production rescan command, opens the bounded replay, performs ten release cycles, samples the complete application process tree, and compares native memory attribution with vmmap. Verification: - Real Issue 272 rendered E2E: PASS - First-open growth: 62.2 MiB within the 400 MiB limit - Ten-cycle measured-tail and backend step growth: 0 MiB - Native versus vmmap difference: 26.6 MiB within tolerance Pre-commit hook ran. Total eslint: 6, total circular: 1
Bring in the develop change that hydrates the initial chat page through block loaders so Issue 443 is tested against the current chat initialization path rather than an obsolete renderer flow. The merge required no replay-specific conflict resolution. Imported sessions remain bounded-replay consumers and native sessions retain the develop block-loader behavior. Verification: - Subsequent SessionCore and chat rendering tests: PASS - Final branch-wide Vitest and Rust gates recorded in the Issue 443 audit: PASS Pre-commit hook ran. Total eslint: 6, total circular: 0
Sampling memory only one second after release confused delayed allocator reclamation with retained replay state and produced a renderer-dominated false failure without proving an actual leak. Keep the 250 MiB acceptance threshold unchanged but sample a bounded 30-second idle window after explicit release. Record the lifecycle rationale and verify that request, watcher, payload, and EventStore ownership remain released while the allocator settles. Verification: - Real Issue 272 idle-release E2E: PASS - Settled growth after the bounded idle window: 0 MiB - No threshold was relaxed to manufacture the pass Pre-commit hook ran. Total eslint: 6, total circular: 0
Integrate the develop runtime incremental-scan work before final acceptance so replay routing, source discovery, session aggregation, and collaboration behavior are evaluated on the same code that will receive Issue 443. Resolve conflicts across Codex indexing, shared session event types, replay commands, source descriptors, SessionCore adapters, and collaboration imports. Preserve develop incremental scanning while retaining the exhaustive 15-source bounded replay contract and native SDE separation. Verification: - Merge-conflict focused Vitest suite: 58 passed - Full Rust and Vitest gates were rerun after conflict resolution Pre-commit hook ran. Total eslint: 6, total circular: 0
Update the issue branch to the develop snapshot that expands usage trends before running the final static, Rust, real SQLite, and rendered Issue 272 acceptance gates. The merge introduced no product conflict in bounded replay. Follow-up test fixtures were updated separately to reflect the current native history-mode and webview observer behavior. Verification: - Relevant post-merge targeted tests: PASS - Full Vitest suite after fixture alignment: PASS - Final real-data gates were rerun on the merged tree Pre-commit hook ran. Total eslint: 6, total circular: 0
The latest develop merge changed native history-mode classification and webview visibility observation, leaving two test fixtures modeled on obsolete behavior. Final reports also needed the current real-data measurements rather than pre-merge evidence. Align the native SDE reconciliation fixture with persisted-database mode, update the webview harness for IntersectionObserver and ResizeObserver, tighten real-session discovery, and record the final architecture and rendered-E2E results. No production behavior is changed by the fixture updates. Verification: - Updated targeted Vitest cases: 7 passed - Full Vitest suite: PASS - Real 3.0 GiB SQLite migration and reopen: PASS - Real 335 MiB Issue 272 rendered E2E: PASS Pre-commit hook ran. Total eslint: 6, total circular: 0
Merge the final develop snapshot that persists the selected organization so the Issue 443 branch is based on the latest shared application state before delivery. The merge required no bounded-replay conflict resolution. The final static checks, Rust suites, real SQLite migration, and rendered Issue 272 lifecycle test were evaluated after this synchronization. Verification: - `pnpm typecheck` and full Vitest suite: PASS - Required Rust package suites: PASS - Real SQLite and Issue 272 rendered acceptance: PASS Pre-commit hook ran. Total eslint: 6, total circular: 0
The audit reports still contained intermediate counts and pre-final-merge wording, which made it difficult for reviewers to distinguish completed acceptance evidence from external infrastructure that remained unavailable. Record the final Vitest count, real SQLite migration and reopen measurements, Issue 272 first-open and lifecycle memory results, native vmmap comparison, and the explicit cloud credential blocker. Keep baseline circular and workspace clippy findings separate from Issue 443 results. Verification: - Final audit evidence cross-checked against completed command output - `git diff --check`: PASS Pre-commit hook ran. Total eslint: 6, total circular: 0
92f2065 to
1f34dae
Compare
|
Reviewed with fable: Verdict: high-quality engineering with four blocking-caliber problems. This is really a re-architecture (~53k production/test lines across 290 files), not a "fix": a new bounded-replay subsystem for 15 external CLI sources, a v10→v12 streamed turn-index rebuild, a cloud-transport rewrite, and a staged collaboration-snapshot ingest. The code is unusually disciplined — parameterized SQL throughout, checked arithmetic, TTL'd/capped registries, an atomic rollback-tested migration, and tests that were strengthened rather than gutted. But five parallel deep-dives (Rust replay, event-pipeline commands, turn-index/CLI, frontend sync, frontend API/UI) plus cross-checks surfaced the following, in priority order. Blocking findings
Major operational risks
Moderate Recommendation |
|
What the PR is actually solving The fix's core idea is small: replace "load everything" with bounded windows (max 10 turns / 200 events / 4 MiB per IPC response), backed by a persistent SQLite index of compact rows plus payload locators — pointers back into the source file so large payloads are range-read in ≤256 KiB slices only when actually viewed. The PR's own numbers: first open of the 335 MB Codex session costs 62 MB once, then repeat opens/polls cost ~0. Why that costs 60k lines ~20k — the new replay subsystem in orgtrack-core. "Bounded" can't be implemented generically because the 15 supported sources store sessions in genuinely different shapes: append-only JSONL logs, Cursor's SQLite KV bubble store, whole-JSON-file formats, Qoder's sidecar. So there are six storage drivers plus the shared index, cache policy, registry, metadata projection, and content-addressed payload artifacts — and each driver must handle incremental sync, torn tails mid-write, UTF-8-safe range reads, and turn-header reconstruction on its own format. The 15-source × per-format-driver matrix is the single biggest multiplier (and, per my earlier pass, carries some avoidable copy-paste between drivers). |
|
Worth requesting on this PR
utf8_boundary_at_or_before/after — identical copies in jsonl_driver.rs:1643 and codex_jsonl.rs:1362
Better as follow-ups
Net effect of the pre-merge items: roughly 1.5–2k lines deleted, three findings fixed structurally rather than patched, and the drift-prone format/validation logic reduced to single sources of truth. If you want, I can draft this as review comments on the PR. |
|
Reuse already happens, ad hoc, in three of six drivers: whole_json_driver imports structured_driver::{compact_chunk, range_from_text, rebuild_turns} — the model citizen The three extractions, biggest first
The payoff and the caveat The caveat: don't push past that. The per-format sync bodies for the SQLite/structured/whole-JSON families do genuinely different work (KV bubble traversal vs. table diffing vs. array diffing), and forcing all six under one universal template would be the wrong abstraction. The high-confidence extractions are the ones above, where the code is already identical or already half-shared. |
|
Worth consolidating
replay request epochs — TTL 5 min, cap 64, LRU (external_replay.rs:4814)
Deliberately not consolidating |
- split the external replay app bridge by window, payload, export, cloud, shell, lifecycle, collaboration, and managed-chunk concerns - separate replay storage drivers by source family and move large inline test modules into focused test files - replace stale Cursor-only and misleading legacy terminology with source-neutral bounded-replay names - align replay generation, revision, episode, and request-token terms across Rust and TypeScript - remove dead compatibility aliases introduced during the #443 migration and format all branch-touched files - preserve Tauri commands, wire values, database schemas, parser versions, replay limits, and native SDE behavior No behavior or wire-format changes. Refs #443
…bounded-external-replay # Conflicts: # src-tauri/crates/orgtrack-core/src/sources/codex/app/index.rs # src-tauri/crates/orgtrack-core/src/sources/codex/app/meta.rs # src-tauri/crates/orgtrack-core/src/sources/imported_history/mod.rs # src-tauri/crates/orgtrack-core/src/store/sqlite/schema.rs # src/features/Org2Cloud/org2CloudBackendAdapter.test.ts # src/features/Org2Cloud/org2CloudBackendAdapter.ts Pre-commit hook ran. Total eslint: 1, total circular: 1
Move bounded replay export destination selection into the trusted Rust/Tauri layer so renderer input can only suggest a file name. Harden export writes with existing-parent validation, symlink and directory refusal, exclusive private temp files, fsync, and atomic replacement while preserving the old destination on failure. Classify collaboration snapshot destinations inside the same immediate transaction used for publication. Allow imported sessions, fresh native forks, and existing snapshots, but reject replacement of ordinary native Agent events and keep their data intact. Update the RPC/UI cancellation contract and add focused Rust, schema, and UI tests for traversal stripping, missing parents, symlinks, atomic failure cleanup, and native snapshot refusal. Refs #443 Pre-commit hook ran. Total eslint: 1, total circular: 1
Build one source-neutral KV order after filtering blank, undefined, duplicate, and stale composer headers, then append persisted fallback bubbles in stable database order. Derive event sequences, lazy turn ranges, compact turn headers, source counts, and fallback user-turn boundaries from that same order. Apply the identical contract to Cursor IDE and Windsurf, and bump both adapters to parser version 2 so existing compact indexes rebuild instead of retaining mismatched ordinals. Add a shared dirty-store fixture covering missing rows, duplicate headers, composer lag, fallback user turns, contiguous pagination, stable IDs, and exact turn ranges. The full orgtrack_core suite passes with 472 tests and 4 existing ignored stress cases. Refs #443 Pre-commit hook ran. Total eslint: 2, total circular: 0
- cap JSONL and Qoder sidecar records before allocation and preserve cursors on torn or oversized input - centralize UTF-8 payload ranges so adjusted offsets make progress without skipping later decoded spans - replace correctness fingerprints with length-delimited SHA-256 while preserving compatibility event IDs - reject Qoder sequence overflow and rebuild sidecar cursors under parser version 4 - treat imported-session suffix wildcards literally across replay lookup and managed CLI dedup - share storage-neutral turn folding and add regression coverage for every boundary Refs #443 Pre-commit hook ran. Total eslint: 2, total circular: 0
- release the global sessions writer lock before pinned replay export I/O and revalidate the final source revision before publication - retry and skip individual CLI content-index failures without discarding healthy session updates - serialize Raw Transcript refresh and backward pagination with a shared request token and immutable replay-version guard - scan filtered backward pages with a bounded loop and expose an explicit retry affordance instead of stalling pagination - cover writer-lock release, retry bounds, empty-page progress, and replay-version mismatch behavior Refs #443 Pre-commit hook ran. Total eslint: 2, total circular: 0
- validate replay identities before allocating foreground or prewarm request tickets - share one bounded request registry and atomically publish foreground/prewarm windows under the manager-store-registry lock order - move Attachment V2 framing, validation, encoder/decoder constants, and golden coverage into one authoritative Rust module - remove the duplicate test-only TypeScript attachment codec and reuse the Rust frame contract in collaboration ingest tests - source replay and Cloud transport limits from a shared budget manifest with Rust parity assertions - preserve Tauri commands, JSON fields, database schemas, provider IDs, and native SDE EventStore behavior Refs #443 Pre-commit hook ran. Total eslint: 2, total circular: 0
- admit one bounded pre-V2 Cloud row for streaming Rust verification while keeping new Attachment V2 rows under the 256 KiB physical limit - stream legacy base64 decoding, wire measurement, and page hashing without allocating duplicate session-sized buffers - upload external replay frozen rows to private Storage in bounded batches and retain only compact object references in the renderer - publish complete native and external replay epochs with one atomic rewrite RPC so crashes cannot expose a truncated transcript - keep old epochs authoritative until final publication and leave interrupted uploads unreachable - cover legacy migration, V2 fail-closed behavior, storage capability failure, rewrite conflicts, cancellation, and large atomic rewrites Preserves the existing Cloud RPC names, segment schema, replay source IDs, parser versions, and native EventStore behavior. Refs #443 Pre-commit hook ran. Total eslint: 2, total circular: 0
- store Cloud replay spools in an owner-only ORGII runtime directory and schedule TTL cleanup even when the renderer disappears - retire expired external-history watcher handles without waiting for another acquire while preserving the three-minute lease boundary - seed an isolated Codex JSONL fixture for the default rendered suite and exercise the production bounded open, render, and release path - auto-skip the real Issue 272 memory scenario when its explicit session id is absent while failing fast when that scenario is requested directly - localize Raw Transcript export, payload-range, and bounded-window notices across every shipped locale - log forward-delta fallback before a bounded authoritative snapshot rebuild and share the Rust unit-response schema - remove redundant replay-export references surfaced by Clippy without changing pinned generation semantics Preserves replay commands, wire values, storage schemas, limits, adapter semantics, and native SDE behavior. Refs #443 Pre-commit hook ran. Total eslint: 2, total circular: 0
- apply rustfmt to the remaining branch-touched replay watermark and provider test files - record final Rust, Vitest, circular-dependency, SQLite, and large-session performance results - document the rendered Issue 272 run and Command+5 manual lifecycle verification - refresh the UI audit with the final localized Raw Transcript coverage - preserve replay behavior, wire formats, database schemas, source IDs, limits, and native SDE routing No behavior or wire-format changes. Refs #443 Pre-commit hook ran. Total eslint: 2, total circular: 0
- merge upstream/develop at 2c8d18a - retain bounded replay changes across the automatically merged orgtrack cache, repository identity, fork, and rendered-chat test files - incorporate the latest README and org2AI repository identity updates - preserve all protected artifacts and checkpoint files outside the commit No manual conflict resolution or #443 behavior change was required. Refs #443 Pre-commit hook ran. Total eslint: 2, total circular: 0
ShiboSheng
changed the title
ShiboSheng
changed the title
|
@Harry19081 Thanks for the detailed review. Replying to the blocking/operational findings in this comment. I treated the four blocking findings, the legacy Cloud migration question, and the strongly recommended operational fixes as pre-merge work rather than documentation-only follow-ups. Blocking and major findings
Moderate/minor findings
Deliberate boundaries retained
The final tree has also passed the real 335.2 MiB Issue 272 open/reopen/lifecycle run, the 3.187 GiB SQLite migration/reopen run, 711 Vitest files / 6,426 tests, 482 |
|
@Harry19081 Replying to the scope/blast-radius analysis in this comment. I agree with the framing: the user-visible bug is one invariant — normal external-history operations must never require complete transcript hydration — but enforcing it removes the old full-array contract from every producer and consumer at once. Cloud/collaboration ingest, Fork, Raw Transcript, export, metadata/hover, managed CLI reopen, and EventStore application remain in this PR because leaving any one of them on the full-load API would preserve the same failure class. The implementation was nevertheless reorganized so those concerns are independently reviewable rather than welded into one command file:
The post-review refactor keeps every new production file below 1,500 lines and does not change the public replay wire, provider IDs, parser compatibility, limits, or native SDE behavior. |
|
@Harry19081 Replying to the structural pre-merge recommendations in this comment. The high-confidence consolidations were implemented, including the ones that fixed live bug classes:
The recommended file split was also brought into this PR rather than deferred:
No new #443 production file exceeds 1,500 lines. The current largest files are the Codex JSONL driver at 1,370 lines, Qoder sidecar at 1,316, and managed-chunk bridge at 1,301. |
|
@Harry19081 Replying to the additional driver-boundary analysis in this comment. I agreed with the diagnosis that ad-hoc reuse was the problem, but stopped short of forcing every storage family behind one universal runtime trait. What changed:
I did not introduce a trait/object dispatch layer or one callback-heavy The resulting structure is under |
|
@Harry19081 Replying to the consolidation priority list in this comment. The final implementation follows the proposed priority where the semantics were truly identical:
The token/impact cold-catalog behavior remains intentionally index-backed: discovery does not revive a prefix/full transcript parser just to avoid temporary zero values before bounded indexing. That tradeoff is documented and keeps the primary memory invariant intact. The architectural cleanup was completed in |
|
@Harry19081 Review feedback update for the analysis starting at the main review:
Final local evidence on the current
The live dual-instance Cloud sharing/presence matrix remains explicitly BLOCKED, not passed: all 12 scenarios reach the credential gate but require unavailable The PR remains Draft for re-review. |
End-to-end context, review history, and current statusThis comment records the complete path from the original incident to the current Draft so a reviewer does not need to reconstruct it from the issue, commits, audit documents, and earlier review comments. 1. What originally went wrongORGII's external-history path treated a CLI session as one complete in-memory transcript. For a Codex/Claude-style JSONL session, opening or refreshing the session could:
The original 29 MiB reproduction became approximately 47.5 MiB before the later IPC, EventStore, snapshot, and worker copies. WebContent reached approximately 1.27 GiB Physical Footprint. This was not only a five-second active-session polling problem. Inactive or completed external sessions could still trigger the same full hydration through metadata, hover, provenance, Fork, Cloud, Raw Transcript, or export. Three earlier issues remained valid but did not fix this path:
#443 was the separate external-history parse/copy problem. During investigation, a second compounding hotspot was confirmed: the SQLite turn-index rebuild collected all event rows and copied/parsing their large JSON fields before folding turns. A real session reached approximately 8.2 GiB Working Set / 8.6 GiB Private and did not finish its index migration. 2. Why the scope became larger than CodexCodex supplied the reproducible JSONL failure, but a Codex-only byte cursor would leave the same full-hydration API available to every other source and consumer. The registered sources use different physical storage:
JSONL byte offsets are not valid SQLite cursors, and neither model can be applied to a whole-document rewrite. The shared layer therefore had to be a bounded replay contract, while source identity, incremental reads, reset detection, tool pairing, stable event IDs, and payload location remained storage/provider-specific. The issue comments were updated to make two boundaries explicit:
3. What the first implementation changedThe implementation replaced complete transcript hydration with:
Large content keeps one canonical representation:
The full-history consumers were migrated together: normal open/refresh, metadata/hover/provenance, Fork, Cloud, Raw Transcript, JSON/Markdown/CSV export, managed CLI reopen/reconcile, and collaboration snapshot ingest. Native SDE Agent remains on its existing persisted-database/EventStore/FSM path. Tests assert that a native session makes zero external replay RPC calls. The SQLite hotspot was changed independently to stream 4. What the review foundThe first deep review correctly described this as a cross-layer re-architecture and found four blocking-caliber issues:
It also required an explicit migration answer for legacy Cloud rows over 256 KiB and identified major operational problems:
The review additionally found bounded-line, wildcard, Qoder overflow, UTF-8 range, hashing, Raw Transcript race/pagination, request-registration ordering, localization, spool/watcher cleanup, and fallback-observability problems. The subsequent architecture comments recommended:
5. How the review feedback was addressed
The high-confidence common code now lives in We deliberately did not force all storage families behind one universal runtime trait or callback-heavy sync template. Mechanical safety invariants are shared; provider normalization, Codex pending/Shell state, SQLite/WAL/KV traversal, Cursor/Warp lineage, and Cline whole-document publication remain explicit adapters. The original 10,638-line application bridge and 3,636-line collaboration ingest file were split by responsibility. No new #443 production file exceeds 1,500 lines; the largest is 1,370 lines. 6. How the complete result was verifiedAutomated/current-tree evidence:
Real-data/performance evidence:
The original real databases and external CLI files were never committed and were tested through isolated/read-only copies or roots. 7. Current state
One external acceptance item is still explicitly BLOCKED, not passed: all 12 dual-instance Cloud sharing/presence scenarios reach their live credential gate but require unavailable The remaining reviewer decision is therefore not whether the known blocking code findings were ignored—they were fixed and tested—but whether the final architecture, deliberate adapter boundaries, and documented external Cloud credential blocker are acceptable for this PR. Refs #443 |
2 participants
Summary
Validation
Known external or develop baselines
Relates to #443