Conversation
Session month shards decode incrementally (one turn at a time, keeping only in-range turns with the exact kept/dropped/carry contract); shard loads run serially with the pre-lock snapshot released before the canonical reload. Out-of-range turns contribute raw dedup keys to cross-file suppression markers; retained strings detach from tokenizer buffers. The codex result cache rewrite, digest wiring, and aggregate parse mode follow as stacked PRs.
|
This one is ready. Thank you for pulling it out. I verified the digest end to end. It is a full 256-bit SHA-256, not truncated: 32 chars, max code point 243, so a one-byte string at half the footprint of hex. It round-trips through I checked every place a digest could meet a raw key. For the no-double-hashing claim I did not want the audit alone, so I ran a cold parse of a real corpus twice, once with workers (27 files through threads) and once with Leaf diff against #1508 is identical on all five periods, cold and warm. tsc clean, 70 touched tests pass. Cost is 0 to 5% wall. On the 52 files: that is the right amount of churn and I am not asking you to reduce it. The plain class is what forces each provider signature to change, which is the compile-time safety we wanted. Two small things I am happy to patch myself: Blocked only by its parents. Please rebase after 0.9.25. |
Per-string flattening at assembly copied every string of every turn (kept or dropped): ~829k turns x dozens of strings of Buffer + string churn that GC could not retire under a heap cap, dying in the sept shard at ~500MB live. streamShardArrayField takes detachStrings (default true); the filtered loader passes false and flattens only what it keeps (kept turns + meta at fold time, carry refs + first project in finishSlice). (cherry picked from commit 1b30177)
The streaming walk costs 15-20s where JSON.parse needs milliseconds (525MB of shards in ~11s on this corpus); it exists for the one shard past V8's max string length. Shards at or under 256MB now take plain JSON.parse through the same per-record projection (filterShardFile / validateCachedFile); larger ones stream exactly as before. A single readFile is already an atomic snapshot, so the small path needs no fingerprint guard. Byte-identical results on both paths are pinned by the size-gate parity suite (kept, sliced, PR-whole, key-overlap, all-dropped, invalid).
The single result file decodes entry by entry; discovery labels come from calls-free metadata; publishes merge dirty entries; single-flight loads are shared and retry on mid-decode publish, with a global clear epoch.
62e5197 to
9b4e9c3
Compare
…reaming Same size gate as session shards (SHARD_STREAM_GATE_BYTES now lives in shard-stream.ts, shared by both): the timestamp scan reuses the exported retainCodexEntry rule per entry so the paths cannot drift, and the entry load applies the same retain rule in memory. The streaming mid-flush rendezvous test pins the stream path explicitly. Parity between paths is pinned by the new codex size-gate test.
Dropped-key markers and shared dedup sets store 32-character SHA-256 digests (~140MB of raw key characters on the reporter corpus); every insert and lookup hashes uniformly. The set is a plain class so unadapted consumers fail at compile time; Antigravity RPC conversations ride along as paired bare keys.
9b4e9c3 to
62f6e34
Compare
|
|
This branch has not been deployed
2 participants
Stacks on #1508 (review that first; this diff includes it until it merges). Split out of #1506 per review: the digest wiring rides separately from the session-shard work.
What changed
DedupSetis a plain class, not aSetsubclass: iteration yields digests, so unadapted consumers fail at compile time. The providercreateSessionParsercontract takesDedupSetthroughout.providers/antigravity-keys.ts.Measured, not assumed:
createHashcosts the same wall time as the previous inline mixer at ~1M keys, so no weaker bound is needed.Verification