perf: reduce snapshot round memory usage

[zxch3n]

Summary

Reduces peak memory across the large-document round

snapshot import → toJSON → remote update import → local updates → local update export → full snapshot export

while preserving lazy state and making cold full-snapshot export substantially cheaper.

What changed

• Bounded SSTable block cache. The old 1 << 20 entry limit could retain nearly every decompressed block touched by a large document. The cache is now byte-weighted and capped at 4 MiB of decompressed data per table.
• Full snapshot export skips the alive-container walk. Referenced containers are ensured when their creating op or imported internal diff is applied. A full export can therefore flush and export the KV state directly instead of decoding every lazy container and rebuilding the same state.
  • Shallow snapshot export still walks the reachable graph and validates parent metadata.
  • Full export preserves imported lazy state bytes faithfully.
  • Empty mergeable children represented by parent-map markers remain marker-only instead of being materialized just for export.
• Cheaper lazy reads for toJSON. Ephemeral reads no longer create a second temporary wrapper for each decoded lazy container.
• Exact snapshot preallocation. Snapshot::encoded_len reserves the final envelope once, avoiding geometric Vec growth while assembling the result.
• Independent scalar-root import fast path. A causally closed batch of new peers writing scalars to new root maps is validated before apply, then diffed from the empty version instead of replaying the full history. The conservative root-history summary is size-capped and incrementally maintained.
• Per-block checksum validation. External snapshot import validates embedded SSTable block checksums instead of allowing malformed inner blocks to fail later during lazy reads.

Benchmark results

Measured with the 11,387,982-byte (10.86 MiB) .procloud fixture using release WASM builds. Timing values are medians of three adjacent, interleaved runs.

Scenario	main (68587bc5)	Before final export change (3b64511b)	PR (3ea96a86)	PR vs main
Cold import + first full snapshot	131.5 ms	400.1 ms	57.2 ms	56.5% faster
First full snapshot phase only	122.7 ms	390.1 ms	46.9 ms	61.8% faster
Full round with causal update	241.1 ms	588.6 ms	264.1 ms	9.5% slower
Snapshot phase in causal round	65.6 ms	359.4 ms	38.4 ms	41.5% faster
Full round with independent update	6,563.6 ms	589.7 ms	264.9 ms	96.0% faster (24.8×)
Second unchanged full snapshot	17.0 ms	5.51 ms	5.60 ms	67.1% faster (3.0×)

The B4 harness's cached snapshot export also improved from 151.04 µs on main to 106.61 µs on this PR (about 29% faster).

Memory comparison against main

The committed memory bench performs 32 local commits with 4 KiB text payloads, exports the resulting local update, then exports and semantically round-trips a complete snapshot. Both revisions were measured with the same script, fixture, parameters, and fresh worker processes.

Update scenario	main WASM peak	PR WASM peak	WASM reduction	main controlled total	PR controlled total	Total reduction	100 MiB gate
Causal update	161.81 MiB	65.94 MiB	59.3%	188.16 MiB	92.29 MiB	51.0%	Fail → Pass
Independent update	310.56 MiB	65.50 MiB	78.9%	336.95 MiB	91.85 MiB	72.7%	Fail → Pass

Controlled total includes the input snapshot/update buffers, WASM linear-memory high-water, returned update and snapshot buffers, retained toJSON heap growth, and retained binary backing stores.

The phase samples show where the WASM reduction comes from:

Phase	main	PR	Change
After snapshot import	23.56 MiB	23.56 MiB	unchanged
After first toJSON	82.00 MiB	38.69 MiB	−52.8%
After causal update import	86.56 MiB	40.75 MiB	−52.9%
After independent update import	252.06 MiB	40.31 MiB	−84.0%
Full snapshot export growth, causal	+74.38 MiB	+22.31 MiB	−70.0%
Full snapshot export growth, independent	+58.50 MiB	+22.31 MiB	−61.9%

Observed process max RSS also fell from 256.34 to 181.13 MiB in the causal scenario (−29.3%), and from 404.38 to 180.42 MiB in the independent scenario (−55.4%). RSS is not the gate because it also includes V8, code pages, and allocator behavior.

Validation

• Release WASM build and package tests: 331 Vitest tests passed (2 skipped, 2 todo), plus 4 Deno and 4 Bun tests.
• cargo test -p loro-internal --lib: 318 passed, 4 ignored.
• Mergeable-container targeted tests: 36 internal tests and 14 public-API tests passed.
• Both memory scenarios pass the 100 MiB controlled-total gate.
• Local-update and full-snapshot round trips preserve semantic versions and JSON state.
• The fixture's byte encoding of the snapshot round-trip version vector differs on both main and this PR; semantic version comparison passes, so this is not introduced here.

Reproduction

pnpm -C crates/loro-wasm build-release
pnpm -C crates/loro-wasm test:snapshot-memory -- /path/to/file.procloud 100 32 4096
cargo run --release -p examples --example b4_bench

Behavior and review notes

• Full snapshot export now relies on the write/apply-time container-materialization invariant. It no longer uses export as a second validation pass over all lazy parent metadata; shallow export retains that validation.
• Per-block checksum validation adds mandatory import work for untrusted bytes. This is a deliberate correctness trade-off.
• The causal full-round result is 9.5% slower than main even though its snapshot phase is 41.5% faster; the table keeps that remaining non-export overhead visible.
• The changeset documents the observable loro-crdt behavior changes, and context/internal-encoding.md plus context/mergeable-containers.md document the maintained invariants.

[github-actions]

WASM Size Report

• Original size: 3077.02 KB
• Gzipped size: 1019.59 KB
• Brotli size: 714.88 KB