perf(encoder): store page index in chunked slices, not a map

[corylanou]

Summary

Encoder kept its page index in a map[uint32]PageIndexElem and sorted the keys on Close. EncodePage already rejects any page number that is not strictly greater than the previous one, so the map never provided anything a slice does not: entries can be appended in write order and streamed out directly.

The map is the dominant memory cost of encoding a large file. One entry per page at roughly 90 B live (plus rehash spikes while the map grows) is ~2.9 GiB for a 34M-page snapshot — the main linear term in benbjohnson/litestream#1477, where a 130 GiB database was OOM-killed in a 12 GiB container. A packed 24-byte entry is ~0.8 GiB for the same file.

• Entries live in chunks whose capacity doubles from 256 entries up to 64K (1.5 MiB). Memory stays proportional to the pages actually encoded: the small files written on every sync cost a few KiB, nothing is preallocated from the caller-supplied Header.Commit (which Compactor copies from unverified input), and growth never copies the whole index.
• encodePageIndex drops the collect-and-sort pass.
• Wire format is unchanged. TestEncoder_PageIndexGolden pins the FileChecksum of a sparse non-snapshot file and a contiguous snapshot, captured from the map-based encoder on main (the checksum covers the index tuples in order, so any reordering or offset drift fails it). TestEncoder_PageIndexChunkBoundaries decodes an index spanning several chunks.
• PageIndexElem and the decoder API are untouched.

Evidence

Measured with the litestream-soak snapshot-compaction-overlap rig (corylanou/litestream-soak#196): Litestream main (3b468c9) built against ltx v0.5.2 versus this branch, same fixture, runtime.MemStats sampled at 100 ms with GOGC=25, heap profile written at each phase's peak. Heap growth is peak HeapInuse minus the phase's post-GC baseline.

ltx	DB	initial L0 sync	snapshot	L1 compaction
v0.5.2	1 GiB / 262,803 pages	35.3 MB (134 B/pg)	69.0 MB (263 B/pg)	101.3 MB (386 B/pg)
this PR	1 GiB / 262,803 pages	9.1 MB (35 B/pg)	48.0 MB (183 B/pg)	77.0 MB (293 B/pg)
v0.5.2	4 GiB / 1,051,216 pages	137.4 MB (131 B/pg)	168.6 MB (160 B/pg)	306.3 MB (291 B/pg)
this PR	4 GiB / 1,051,216 pages	31.4 MB (30 B/pg)	70.3 MB (67 B/pg)	204.3 MB (194 B/pg)

Each phase includes a fixed 32 MB of S3 multipart buffers (part size × concurrency+1), so the per-page deltas are larger than the totals suggest: the encoder index goes from ~95 B/page to ~25 B/page. Peak heap profiles (inuse_space, 4 GiB snapshot phase):

v0.5.2:   99.23MB ltx.(*Encoder).EncodePage
           32.00MB s3/manager.(*maxSlicePool).newSlice
this PR:  19.04MB ltx.(*pageIndex).append
           32.00MB s3/manager.(*maxSlicePool).newSlice

Combined with the per-database maintenance serialization in the companion Litestream PR (linked in a comment below), the #1477 overlap drops from ~450 B/page on today's code to ~195 B/page (4 GiB fixture: 470.6 MB → 205.6 MB), i.e. ≈15 GB → ≈6.7 GB at the reporter's 34.2M pages. With that change alone the biggest remaining per-page cost is the decoder-side map noted below (DecodePageIndex, 81 MB in the 4 GiB compaction profile).

Follow-up (not in this PR)

The decoder side (DecodePageIndex / Decoder.PageIndex) still returns a map[uint32]PageIndexElem, which is now the largest remaining per-page term during compaction (~70 B/page). Changing that is a public API change and belongs in its own PR.

Test plan

• go test ./..., go vet ./..., gofmt
• Golden checksums captured from main (8cb8f8e) before the change and asserted after
• Chunk-boundary round trip (2×65,536 + 259 pages)
• Rig A/B above at 1 GiB and 4 GiB
• Two adversarial Codex review passes; the preallocation-from-header finding from the first pass drove the chunked design

[corylanou]

Companion Litestream change (per-database maintenance serialization, the other half of the #1477 fix): benbjohnson/litestream#1479

[corylanou]

Evidence for the combined change, built by the soak rig from the litestream PR head with go.mod pinned to this branch (d457a1a), no local replace: 4 GiB / 1,051,216-page fixture, snapshot held resident while L1 compaction runs — overlap peak heap growth 470.6 MB on litestream main + ltx v0.5.2 → 199.6 MB with benbjohnson/litestream#1479 + this PR (snapshot phase 168.6 → 68.8 MB; encoder index 99 MB → 16 MB in the peak profile). Full table and profiles in the litestream PR body.