Nightly 1400 backend exhausts heap during attachment cleanup on startup

[Gigioxx]

What happened

The installed macOS T3 Code Nightly stopped starting after updating to 0.0.41-nightly.20260908.1400. Its embedded backend repeatedly exhausted the JavaScript heap and aborted. Restoring official signed 0.0.41-nightly.20260908.1387 restored access to the same existing projects and threads without deleting history or editing projection cursors.

Diagnosis

The attachment-cleanup bootstrap added by #9871, commit 7220dfe2c949476eaa7d21eccbcd3a0ce0eddb49, forces a full historical event replay when its new cursor is absent or zero, even when all ordinary projections are caught up.

On the stopped-app recovery backup:

• All nine ordinary projection cursors were at event 1131083; projection.attachment-cleanup was 0.
• The event store held 1,131,083 events and 3,441,699,990 serialized payload bytes. The largest event was 67,627,295 bytes; the largest 500-event page was 95,024,354 bytes.
• Only 24 thread.deleted events, totaling 2,160 payload characters, were relevant to cleanup. There were no thread.reverted events.

ProjectionPipeline.ts persists the starting cursor, runs readFromSequence(cleanupStart, Number.MAX_SAFE_INTEGER), filters deletions/reverts in JavaScript, and advances the cleanup cursor only after the complete scan and successful cleanup. An OOM leaves the next launch starting from zero again.

The existing event-store reader uses 500-row SQL pages but recursively concatenates them inside Stream.flatMap. A synthetic comparison using Effect 4.0.0-rc.112 reproduces retention of consumed pages even with a discard consumer. Non-recursive Stream.unfold pagination over the same generated rows stays flat in that test.

The native crash stack reaches node::sqlite::StatementExecutionHelper::All; the Node SQL adapter calls statement.all. This is paged SQL with cumulative retention, not a single SQL query returning the entire database. The shipped 1400 ASAR contains the same cleanup and recursive reader code.

The evidence strongly connects the newly forced full replay to a pre-existing stream-retention problem. A full-app heap profile has not been captured, so the exact retained-object graph and contribution of other startup allocations remain unproven.

Steps to reproduce

Observed on the affected profile, not repeated against live data during this diagnosis:

1. Start with an established 1387 profile containing a large orchestration event history, with ordinary projections caught up.
2. Update the desktop app to Nightly 1400 and launch it.
3. Observe the embedded backend abort with V8 OOM during startup and retry without advancing the cleanup cursor.
4. Restore 1387; the existing environment becomes usable again.

The portable synthetic test below can be saved as stream-retention.mjs. It uses 100 pages of 500 generated events with 2,048-character text payloads, a discard consumer, forced GC every 10,000 events, and count assertions. Run against an existing Effect 4.0.0-rc.112 package:

node --expose-gc stream-retention.mjs recursive /absolute/path/to/effect
node --expose-gc stream-retention.mjs flat /absolute/path/to/effect

stream-retention.mjs

import { pathToFileURL } from "node:url";
import path from "node:path";
const effectRoot = process.argv[3];
if (!effectRoot) throw new Error("Pass variant (recursive|flat) and absolute path to effect package directory");
const Effect = await import(pathToFileURL(path.join(effectRoot, "dist/Effect.js")));
const Stream = await import(pathToFileURL(path.join(effectRoot, "dist/Stream.js")));
import assert from "node:assert/strict";
const pages = 100;
const pageSize = 500;
const readRows = (cursor) => Array.from({length: pageSize}, (_,i) => ({sequence: cursor+i+1, payload: JSON.parse(JSON.stringify({text: String(cursor+i).padEnd(2048,"x")}))}));
const recursive = (cursor, remaining) => Stream.fromEffect(Effect.sync(() => readRows(cursor))).pipe(Stream.flatMap(events => {
 const nextRemaining = remaining - events.length;
 if(nextRemaining <= 0) return Stream.fromIterable(events);
 return Stream.concat(Stream.fromIterable(events), recursive(events.at(-1).sequence,nextRemaining));
}));
const flat = Stream.unfold(0, cursor => Effect.sync(() => cursor >= pages*pageSize ? undefined : [readRows(cursor), cursor+pageSize])).pipe(Stream.flattenIterable);
let count=0;
await Effect.runPromise(Stream.runForEach(process.argv[2] === "recursive" ? recursive(0,pages*pageSize) : flat, () => Effect.sync(() => {
 count++;
 if(count % 10000 === 0) { global.gc(); console.log(JSON.stringify({variant:process.argv[2],count,heapMB:Math.round(process.memoryUsage().heapUsed/1024/1024)})); }
})));
assert.equal(count,pages*pageSize);

Both consume exactly 50,000 events. This reproduces stream retention only; it is not a minimal full-application reproduction. No private database or event contents are attached.

Version

Failed: 0.0.41-nightly.20260908.1400, release commit eb115063634c416c6362cc407f8572cb0c136ddf.

Working rollback: 0.0.41-nightly.20260908.1387.

Environment

macOS 27.0 (26A5425a), arm64, desktop app with its embedded local Electron backend. Triage launched with npx t3@nightly triage; its Node v24.15.0 describes the triage process, not a verified version of the failed Electron Node runtime. Source lockfile pins Effect 4.0.0-rc.112.

Evidence

2026-09-08T13:01:20.803Z
Mark-Compact (reduce) 3693.7 (3751.1) -> 3693.5 (3749.1) MB
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

2026-09-08T13:03:01Z macOS crash report, version 1400:
EXC_CRASH (SIGABRT), Abort trap: 6
node::OOMErrorHandler
v8::String::NewFromUtf8
node::sqlite::StatementExecutionHelper::ColumnToValue
node::sqlite::ExtractRowValues
node::sqlite::StatementExecutionHelper::All

Synthetic retained heap after forced GC (MiB):
events       10000  20000  30000  40000  50000
recursive      29     50     71     91    112
unfold         10     10     10     10     10

Desktop traces record backend restart scheduling after OOM. The source awaits projection bootstrap before starting the orchestration worker. The 1387-to-1400 comparison changes the cleanup path but not the event-store implementation or schema migrations.

Related issues

No confirmed duplicate found in targeted searches on September 8, 2026. #8648 concerns long-uptime OOM on 0.0.35; #7075 concerns provider resume payloads. Neither establishes this new cleanup-startup trigger.

Merged #8992 bounds WebSocket replay payloads, but its guard does not cover cleanup bootstrap. Open #10512 addresses a V2 compatibility RPC rather than this startup path. #7537 documents why silently capping projection catch-up is unsafe.

At investigation time 1400 was the newest published release. Main at 061543e9e5b54ec0048725c37d52fef2962df173 had no subsequent cleanup fix.

Fix applied or workaround

Recovery restored official signed Nightly 1387 and verified existing projects and threads were accessible. User history and attachments were preserved. This triage only queried the recovery backup read-only and inspected source/logs; it made no live changes.

Suggested upstream direction: non-retaining pagination in the shared event reader, plus a cleanup query that reads only necessary deletion/revert metadata. Preserve full catch-up and retry-safe cleanup semantics. Raising the heap limit or manually skipping cleanup is not a verified fix.

Filed by

Diagnosed and filed with GPT-6 Astra in the Codex harness via T3 Code triage, including a synthetic retention experiment and read-only database measurements.

[joshfcc]

Same issue here on a separate Mac. I updated this morning before resuming my sessions and couldn't get a usable window open. Rolling back to the official signed 1387 build restored access to the same history.

Codex helped investigate locally. Confirmed details:

• macOS 26.5.1 (25F80), Apple M4 Pro, 24 GB RAM, arm64. This also affects a non-beta macOS release.
• Failing version: 0.0.41-nightly.20260908.1400, bundled Electron 44.1.0. The installed app passed strict deep signature verification.
• Backend repeatedly aborted after roughly 25-40 seconds, with JavaScript heap exhaustion around 3.7 GB. The native stack includes node::OOMErrorHandler and node::sqlite::StatementExecutionHelper::All. Local HTTP requests timed out during the crash loop.
• Database integrity check: PRAGMA quick_check = ok. About 7.9 GiB, 1,265,321 events, 651,077 activities, 400 threads, 31,306 messages, and 3,103 turns.
• All nine existing projection cursors were at 1,265,321. The new projection.attachment-cleanup cursor was 0. Installed source and traces match the full-history cleanup replay described above.

After backing up the database and WAL, we restored 0.0.41-nightly.20260908.1387 without deleting history or editing cursors. The window and an existing conversation loaded, including its latest image attachment. Thread, message, and event counts were unchanged. HTTP returned 200 across repeated checks, and the same backend process stayed alive through a 90-second check at roughly 300-350 MB RSS. That's a startup recovery check, not a long-running soak test.

This independently confirms the startup regression and rollback workaround. We haven't captured a heap profile or independently tested the stream-retention explanation in this issue. No private database or conversation contents are attached.

The recovered app still offers 1400 for download, so it's easy to reinstall the broken release. Please flag the affected Nightly while the fix is being worked on.

[jmqcooper]

Third data point, plus two things not yet covered in this thread: 1414 is also affected, and advancing the stuck cursor in place fixes it on the broken build — which isolates the cleanup replay as the cause rather than something merely correlated with it.

1414 is affected too

Both reports above are on 1400. I hit the identical crash loop on 0.0.41-nightly.20260908.1414, so the regression carried through to the newer nightly. Worth folding into the request to flag affected builds.

• macOS 26.5.2 (25F84), Mac16,8, 24 GB RAM, arm64; Electron 44.1.0
• Backend child aborted ~20–28 s after spawn, six crash reports in three minutes, OOM at ~3.4–3.5 GB
• Same signature: node::OOMErrorHandler → v8::ValueDeserializer::ReadValue, faulting thread named V8Worker
• Nine ordinary cursors at 1329758, projection.attachment-cleanup at 0 with its initial 1970-01-01T00:00:00.000Z timestamp — never once committed

Store shape, for comparison with the two above:

	this profile	@Gigioxx	@joshfcc
events	1,329,758	1,131,083	1,265,321
payload_json bytes	3,976,331,322	3,441,699,990	—
largest single event	27,281,765	67,627,295	—
threads	1,095	—	400
messages	65,049	—	31,306
activities	1,072,627	—	651,077
thread.deleted	217	24	—
thread.reverted	3	0	—

Both prior reports had zero thread.reverted events. This profile has three, so the revert branch of applyAttachmentSideEffects (and the if (!cleaned) return; early-return that deliberately holds the cursor back) is exercised here too, not just the delete branch.

Advancing the cursor in place confirms the diagnosis

Rolling back to 1387 restores access but doesn't isolate the cause. Instead I stayed on the crashing 1414 build and changed one row — the only difference between runs:

UPDATE projection_state
SET last_applied_sequence = (
      SELECT MIN(last_applied_sequence) FROM projection_state
      WHERE projector <> 'projection.attachment-cleanup'
    ),
    updated_at = strftime('%Y-%m-%dT%H:%M:%f','now') || 'Z'
WHERE projector = 'projection.attachment-cleanup';

Same binary, same database, same machine:

	before	after
event-log read batches per boot	8,179	5
backend RSS	climbs to ~3.5 GB	flat ~217 MB
backend PID	new one every ~25 s	stable, 5+ min uptime
crash reports	6 in 3 min	0

Counted from ~/.t3/userdata/logs/server.trace.ndjson by tallying sql.execute spans whose db.query.text contains eventId. The 5 remaining batches are the ordinary incremental catch-up.

Since cleanupStart is the only thing that changed, this is direct confirmation that the full-history replay from sequence 0 is what exhausts the heap — consistent with the retention analysis in the issue body. Data was unaffected: thread/message/activity counts identical before and after, PRAGMA quick_check clean.

The trade-off is real, so this is a stopgap and not a fix: attachment files for the 220 already-deleted/reverted threads are never reclaimed, because the scan that would have found them is skipped.

More heap is not available as a workaround

Worth ruling out explicitly, since it's the obvious first thing to try: NODE_OPTIONS=--max-old-space-size=... cannot reach the backend.

• Electron strips NODE_OPTIONS in packaged apps. Launching the binary with it set and inspecting the child with ps -Eww shows it arriving as NODE_OPTIONS= — present but empty.
• The backend child spawn hardcodes execArgv: [].
• The bundle exposes no --js-flags or T3CODE_* memory passthrough.

So on a store past the threshold there is no supported way for a user to give the replay enough heap to complete even once. The only routes are a rollback or a manual database edit, neither of which is discoverable from a crash dialog.

Minor provenance note

Consistent with #9871 being the origin: the string projection.attachment-cleanup is absent from the shipped bundles of 0.0.28 and 0.0.29-nightly.20260708.754, and present in 0.0.41-nightly.20260908.1414. The first release containing #9871 is v0.0.41-nightly.20260908.1400.

No heap profile captured here either, so I can't independently confirm the retained-object graph.

[alecramos-sudo]

Fourth data point, on a larger store and older macOS, plus a longer time-to-crash that fits the replay explanation.

• macOS 15.7.3 (24G419), M1 Max, 64 GB RAM, arm64. So this is not a RAM-constrained machine; it's the V8 heap cap.
• Failing version: 0.0.41-nightly.20260908.1414. Backend aborted ~72-82 s after spawn (two runs), later than the 20-40 s reported above, which tracks with the store being ~3x larger.
• Same signature: FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory, node::OOMErrorHandler → node::sqlite::StatementSync::All, heap at ~3.85 GB on the last GC. Desktop shell then hit Timed out after 60000ms waiting for desktop backend readiness.
• Migrations 48_ProjectionThreadBranchPullRequest and 49_ProjectionThreadsActiveOrderKey ran fine immediately before; they're just ALTER TABLE ADD COLUMN.

Store shape:

	this profile
DB size	13.4 GB (3,280,665 pages × 4096, freelist 0)
events	4,019,125
payload_json bytes	4,399,700,862
largest single event	9,601,897
threads	1,037
messages	86,344
activities	948,091
thread.deleted	155
thread.reverted	0

Nine ordinary cursors at 4019125; projection.attachment-cleanup at 0 with the 1970-01-01T00:00:00.000Z timestamp. attachment-cleanup is absent from 0.0.36-nightly.20260828.1208 and 0.0.32, present in 1414.

Rolled back to the signed 1387 zip (no DB or cursor edits). Backend answered readiness with 200, stayed alive through a 90 s check at ~330 MB RSS, history intact. Not a soak test.

Two small notes for whoever picks up #10777:

1. The largest event here is under 10 MB, so the OOM does not need any giant single payload; total retained volume is enough on its own.
2. On a store this size even a fixed, non-retaining replay of 4M events from sequence 0 is going to be a long first boot. It may be worth seeding projection.attachment-cleanup at the minimum of the existing cursors when the row is first created, and only doing the historical scan opportunistically or behind an explicit action.