[azank1]

The builder refused buildable repos today, and memory kills took down whole instances. There were three causes, and each one was measured.

From production's function logs (2026-10-13):

Repo	Outcome	What the log read
anomalyco/opencode	too big	1,231 MB live, during its first fetch's indexing
affaan-m/hermes-agent	too big	1,211 MB live, same point
google/dawn	too big	1,244 MB live. Its first fetch read 1,196 MB live there, but 484 MB replayed alone
google/site-kit-wp	busy	299 MB live, 2,083 MB resident

• One instance sat at 1,341 MB resident with 30 to 60 MB live.
• Vercel reports 8 "instance was killed because it ran out of available memory" on /api/build/repo since 2026-10-11. The last was at 13:18 UTC today.

1. Trees held 16 KB each (fda4196)

zlib returns a small inflated object as a view of its 16 KB output buffer. The object store kept trees as they came, so a 300-byte tree held 16 KB. Here are window packs indexed from the same saved bytes:

Window pack	Trees	Held before	Held after	Peak live before → after
anomalyco/opencode, 172 MB	129 MB	684 MB	130 MB	1,041 → 468 MB
affaan-m/hermes-agent, 236 MB	464 MB	766 MB	465 MB	1,151 → 841 MB
google/skia, 29 MB	53 MB	122 MB	53 MB	live after indexing 175 → 106 MB

Kept trees are now copied into shared 1 MB slabs. Indexing opencode's pack got faster: 15.6 s before, 12.2 s after.

2. Too big judged the whole process (b94bfef)

Other requests' memory counted against a replay, as with dawn above. The guard now counts what the replay itself holds: live memory beyond what was live when it began. Each big replay's turn opens with a thorough collection, so that starting point is not another replay's garbage.

3. Pooled pages stayed resident (b94bfef)

After a big replay, V8 keeps the heap pages it freed in a pool, and two normal collections don't give them back. V8's last-resort collection does. Replayed locally, google/dawn left 1,002 MB resident for 57 MB live after two normal collections, and 427 MB after the last resort.

A full instance (resident past 1.9 GB) now gets the last-resort collection, and is busy only if that doesn't make room. Before, for two seconds after any collection, the guard ignored a full instance.

4. Downloads weren't checked (b94bfef)

The guard now runs as each pkt-line arrives. It keeps room for the copy a pack takes when its chunks are joined: 236 MB more for hermes-agent's first pack.

I also tried rewriting the receive loop to copy into doubling blocks. At the 16 KB chunks HTTPS delivers, it measured the same as the current loop: 285 vs 293 MB peak for the 236 MB pack. It's not in this PR.

This branch on a warm instance

Big repos replayed one after another in one local process, under production's limits, each turn opened as buildRepo now opens it. Peak figures are what the guard read, garbage included.

Repo	Production today	This branch	Peak live / resident	After the next turn's collection
google/dawn	too big	4,751 commits, 90 MB of packs	778 / 946 MB	11 MB live, 261 MB resident
anomalyco/opencode	too big	17,222 commits, 202 MB of packs	777 / 959 MB	11 MB live, 297 MB resident
affaan-m/hermes-agent	too big	20,000 commits (the record's cap), 393 MB of packs	1,124 / 1,361 MB	12 MB live, 599 MB resident

hermes-agent took 814 s here, 503 s of it in one fetch of 4,000 blobs over my connection. Production fetches far faster, but its 240 s deadline is the open question for that repo. On main, dawn alone left 1,002 MB resident after two normal collections.

Not fixed here. google/site-kit-wp needs more than the 400 MB pack budget: 394 MB in five fetches, leaving 6 MB for the sixth. It stays too big, correctly.

Tests

• The tree test fails on main: 201 trees, 12,690 bytes, held in 3,293,184.
• The held-memory test fails on main: "replay needs more memory than the hosted builder has".
• The guard's unit tests run on a fake clock and fake memory. They cover too big against the starting point, garbage collected before it's believed, the last resort when full, the reserve, and the read interval.
• A download test splits pkt-lines across 3-byte and 1-byte chunks.
• Suite: 58 pass, 6 live tests skipped. Typecheck is clean.

After deploy:

1. Rebuild google and affaan-m.
2. Read back the new log line, big replay's turn after … collected: … rss X -> Y MB. That confirms the last-resort collection on Node 24; it was measured locally on Node 22.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
commit-city	Ready	Preview	Oct 13, 2026 2:21pm UTC