Conversation
|
The parse-layer work is good and the numbers are clean. On a frozen copy of a real corpus I diffed the full JSON export leaf by leaf against main for today, week, 30days, month and all: zero differences from this PR. The single The memory result is real: Before merge:
|
aa6ef7c to
2682736
Compare
Session month shards decode incrementally (one turn at a time, keeping only in-range turns with the exact kept/dropped/carry contract); shard loads run serially with the pre-lock snapshot released before the canonical reload. Out-of-range turns contribute raw dedup keys to cross-file suppression markers; retained strings detach from tokenizer buffers. The codex result cache rewrite, digest wiring, and aggregate parse mode follow as stacked PRs.
2682736 to
702bd19
Compare
|
The memory result is real and the output is exact. Thank you.
I re-checked the three hazards. In one resident Still open:
Conflicts for the rebase. Against main: |
Per-string flattening at assembly copied every string of every turn (kept or dropped): ~829k turns x dozens of strings of Buffer + string churn that GC could not retire under a heap cap, dying in the sept shard at ~500MB live. streamShardArrayField takes detachStrings (default true); the filtered loader passes false and flattens only what it keeps (kept turns + meta at fold time, carry refs + first project in finishSlice). (cherry picked from commit 1b30177)
The streaming walk costs 15-20s where JSON.parse needs milliseconds (525MB of shards in ~11s on this corpus); it exists for the one shard past V8's max string length. Shards at or under 256MB now take plain JSON.parse through the same per-record projection (filterShardFile / validateCachedFile); larger ones stream exactly as before. A single readFile is already an atomic snapshot, so the small path needs no fingerprint guard. Byte-identical results on both paths are pinned by the size-gate parity suite (kept, sliced, PR-whole, key-overlap, all-dropped, invalid).
The single result file decodes entry by entry; discovery labels come from calls-free metadata; publishes merge dirty entries; single-flight loads are shared and retry on mid-decode publish, with a global clear epoch.
bb2415b to
bfe4f62
Compare
…reaming Same size gate as session shards (SHARD_STREAM_GATE_BYTES now lives in shard-stream.ts, shared by both): the timestamp scan reuses the exported retainCodexEntry rule per entry so the paths cannot drift, and the entry load applies the same retain rule in memory. The streaming mid-flush rendezvous test pins the stream path explicitly. Parity between paths is pinned by the new codex size-gate test.
Dropped-key markers and shared dedup sets store 32-character SHA-256 digests (~140MB of raw key characters on the reporter corpus); every insert and lookup hashes uniformly. The set is a plain class so unadapted consumers fail at compile time; Antigravity RPC conversations ride along as paired bare keys.
overview -p today (all-provider, unfiltered) strips per-call payloads the headline never reads (tool sequences, dedup keys, per-call paths, name arrays; shell commands extracted first for PR launch matching) per provider right after parsing, so peak tracks one provider instead of the corpus. Denylist strip, single narrow-shape gate in the durable builder, memo bypassed both ways. Byte-identical output proven by lite-parity suite.
project/prLinks ride on calls outside the ParsedApiCall type and survive a pure type-keyed denylist; they are heavy, so they drop explicitly. Covered by extended strip assertions.
Tool-name strings like 'Bash' repeat ~1.5M times as distinct objects (~90MB heap); the tool breakdown is precomputed before stripping and the edit-time median is unrendered here. Model, provider, speed, category, and project names intern to one shared ref per provider batch.
userMessage averages ~115B per turn but totals ~40MB serialized (~100MB heap); launch matching needs only the first qualifying 160-char prefix. Precompute turn.promptPrefix at strip time and drop the full text; the matcher reads the prefix first with the userMessage fallback, so attribution is identical. Correction scans tolerate empty messages (their stats are unrendered here).
Per-provider stripping left the provider's full summaries resident through its whole parse (death in early provider parsing at cap). buildSessionSummary now strips each session the moment it resolves under an AsyncLocalStorage-gated per-parse store (safe for concurrent mixed parses), so peak never holds more than one session's full turns.
A complete-cache refresh streamed every in-scope shard twice: once before lock acquisition and once after. The pre-lock snapshot stays referenced until the reload replaces it, so peak held two filtered copies plus a 730MB shard transient and died near 500MB on overview -p today. Every saveCache mints a fresh envelope nonce, so an unchanged nonce proves byte-identical shards and the reload is skipped; any publication, torn read, or missing envelope keeps the reload.
bfe4f62 to
4dc5436
Compare
|
Body fixed: userMessage is emptied, matcher reads the precomputed 160-char promptPrefix first with fallback. The dropped comment closing line is restored (folded into the commit that dropped it). 1b30177 is out of this stack (lives in 1506 now). Timings in body: strip trades about 9s for about 1GB less peak. Full suite plus locks green. Rebase after 0.9.25; noted the warp and usage-aggregator semantic conflicts for then. |
2 participants
Fixes #1505 (stacked on #1509; restacked onto the split stack).
Stacks on #1506 (review that first; this diff includes it until it merges). Fixes the remaining
codeburn overview -p todayheap crash at the parse layer.What was wrong
After the load-phase bounds from #1502, the same-day parse still retained ~1-2GB: every provider's full turns co-resident through merge, day aggregation, and rendering. Capped at 512MB the command still exited 134; the heap floor measured between 512MB and 1024MB.
What changed
stripForAggregateparse mode: per-call payloads the headline never reads (tool sequences, dedup keys, per-call paths, bash/skill/mcp/subagent arrays) are stripped per session insidebuildSessionSummarythe moment it resolves (in place, single-owner — the shared memo is bypassed both ways, so peak never holds more than one session's full turns).call.commands); PR launch matching reads them through a derivation-equivalence-tested helper. userMessage is emptied and replaced with a precomputed 160-charpromptPrefix(the full text averaged ~115B/turn but totaled ~40MB serialized / ~100MB heap); the matcher reads the prefix first with the userMessage fallback, so attribution is identical.saveCachemints a fresh nonce, so an unchanged nonce proves byte-identical shards and re-streaming them only doubled peak heap for zero new data. Any publication, torn read, or missing envelope keeps the reload (pinned byparser-cache-refresh-reload-skip.test.ts).The detach-only-retained-strings and size-gate commits now live down the stack in #1506, so no step here regresses on its own.
Verification
overview -p todaycompletes with real data (was:FATAL ERROR: JavaScript heap out of memory, exit 134); lite-parity suite proves byte-identical rendered output and equal totals/days/models/categories on an isolated corpus.Timings (warm cache, medians of 3, 1.2GB shard corpus with one 697MB shard)
overview -p todayreport --period today --format jsonThe strip trades ~9s for ~1GB less peak on this corpus (the lite sessions replace full turns through merge/aggregate/render).
reporttakes the full-session path and still benefits from the gate plus the skipped reload below it.Notes for review
toolSequence/spawn ids/paths/name arrays/dedup key out, everything else spread through), so futureParsedApiCallfields stay populated.main.tsrequests lite unconditionally foroverview;buildDurablePeriodis the only decider (today-only, all-provider, no filters/selection).servedispatchesoverviewin-process, so the strip also runs in the resident desktop/menubar process, not only one-shot CLI runs. Verified: a liteoverview -p todayfollowed by a fullreport --period today --format jsonin oneserve --stdioprocess returns identical full data (memo bypass is per-parse, not process-global).