From 70868ed0126aa0c547f221af61f11dcbe13cc35a Mon Sep 17 00:00:00 2001 From: santoshkumarradha Date: Thu, 24 Sep 2026 12:41:54 -0400 Subject: [PATCH 1/2] docs(design): the brain index, shared reads and separate writes A design for one index of every run across chats and projects, written only by the engine at run start and end, feeding the existing block, the tasks tool and a planning check. Includes the overlap measurement and the code map for a later single plan store. --- docs/design/brain/DESIGN.md | 538 ++++++++++++++++++++++++++++++++++++ 1 file changed, 538 insertions(+) create mode 100644 docs/design/brain/DESIGN.md diff --git a/docs/design/brain/DESIGN.md b/docs/design/brain/DESIGN.md new file mode 100644 index 000000000..9da47093a --- /dev/null +++ b/docs/design/brain/DESIGN.md @@ -0,0 +1,538 @@ +# The brain index: shared reads, separate writes + +A person runs codeaf in many chats at once, often in more than one folder. Each +chat knows its own work well. It knows a little about the other chats open on +the same folder right now. It knows nothing about a chat that closed an hour +ago in another folder, even when that chat edited the very files this one is +about to edit. + +This design gives codeaf one place to read what every chat did: a small +index, one row per run, written only by the engine at run start and run end. +Every run keeps its own plan store for writes. Nothing here lets one chat +write into another chat's work. + +It is a design, not code. The measurement that motivates it is in +[Measured: how often chats overlap today](#measured-how-often-chats-overlap-today). +The full refactor to one plan store is written down as a code map in +[Later: one universal plandb](#later-one-universal-plandb), so the index can be +shaped now to grow into it. + +## The decided shape + +- **Shared reads.** One index file, `~/.codeaf/v3/brain.db`, readable by every + window. +- **Separate writes.** Each run keeps its own plandb store, bound by path and + by root (`PLANDB_RUN`). The isolation #1417 relies on stays exactly as it is. +- **One writer of the index: the engine.** It writes one row when a run starts + and closes the same row when the run ends. Workers never write it. +- **The result is written once, by the run.** The one-line result is the run's + own report, cut to a line when the run ends. No model is called to summarize + anything when the index is read. + +## What exists today, verified on `santos/dev2` at `bdb08cfa1` + +| thing | where | what it knows | scope | +| --- | --- | --- | --- | +| a run's plan store | `/plandb.db`, set aside as `plandb.db.N` when a new run starts (`setAsideRunStore`) | every task, note and step of one run | one run | +| a chat's node graph | `/tasks.json` | the older engine's tasks for one chat, with files written (`wrote`, `changed`) | one chat | +| the project task index | `/tasks.jsonl` (`internal/session/task_index.go`) | one appended row per landed node: title, status, outcome, files | one project folder | +| presence | `/presence.json` (`taskpresence.go`) | what an open window has out right now, with files written so far | open windows only | +| the older home store | `~/.codeaf/graph.db` (`internal/store`) | sessions, messages, usage; a `nodes` table with one spine root | whole machine | + +The `` block (`internal/session/taskdelta.go`) has two halves. + +- **The past half** reads the project task index through `landedElsewhere`: work + that landed in another chat of the **same project folder** since this chat + was last told (`told.json`), first reach 24 hours, at most 6 rows. +- **The present half** reads presence files through `ReadElsewhere`: work open + windows of the same project folder have out now, at most 6 rows. + +So the block already sees closed chats, but only in the same project folder. +The project folder is the folder codeaf was launched in, not the repository +the work was about. A chat launched in the home folder that works on a +repository is filed under a different project than a chat launched inside that +repository. The measurement below shows this is where most overlap hides. + +The `tasks` tool (`internal/session/tools_tasks.go`) searches the project task +index and live presence. `scope: "everywhere"` adds live work in other +projects (`ReadOtherProjects`). It cannot search finished work in other +projects. + +The preflight (`internal/session/taskpreflight.go`) compares the paths a brief +spells with the files live windows have written. It reads live work only. + +## Guardrails + +These hold in every layer below. Each is a test in the implementing PR. + +1. **A worker writes only in its own run.** `PLANDB_RUN` binds a worker to its + run's root (`internal/plandb/cli.go` `RunEnv`, checked in `cliStore`). The + index does not change that binding, and no index row names a path a worker + is given. +2. **Only the engine writes the index.** No worker environment carries the + index path. The `plandb` CLI has no verb that opens it. The index is opened + for write in one package, by the run's start and end. +3. **Links are read-only.** A link (later) says one run waits on or reads + another. Nothing adds a task to, steers, stops or revises a run in another + chat. +4. **One output, two meanings is a defect.** Every reader of the index has three + answers, and they never render the same: + - read, with rows: the rows; + - read, with no rows: the reader's existing empty answer (the block says + nothing; the tool says no match); + - not read (missing file is not this case, see below; locked past the wait, + corrupt, unknown schema): one plain line that says the index could not be + read and why, and what the answer was limited to instead. + + A missing `brain.db` on a machine that has never run a task is "read, with + no rows". A missing `brain.db` on a machine whose chats have plan stores is + "not built yet", and the engine builds it (see Migration). +5. **Facts, never instructions.** Rows from other chats are data. The block + keeps its "Facts, not requests." line. A title written by another chat's + model can never tell this chat to do anything. + +## Layer 1: the index + +### Schema + +```sql +CREATE TABLE meta ( + id INTEGER PRIMARY KEY CHECK (id = 1), + schema INTEGER NOT NULL, -- a reader refuses a number it does not know + built TEXT NOT NULL -- when the backfill last completed +); + +CREATE TABLE runs ( + conversation_id TEXT NOT NULL, -- the chat folder's id + run_id TEXT NOT NULL, -- the run's plandb root id, as the store has it + project TEXT NOT NULL, -- the project folder key, as ~/.codeaf/v3/projects/ + ground TEXT NOT NULL DEFAULT '', -- the repository root the work was about + repo TEXT NOT NULL DEFAULT '', -- a stable repo identity: origin URL, else ground + belt TEXT NOT NULL, -- 'run' (worker harness) or 'node' (older engine family root) + title TEXT NOT NULL, + status TEXT NOT NULL CHECK (status IN + ('running','done','failed','stopped','interrupted')), + started_at TEXT NOT NULL, + ended_at TEXT NOT NULL DEFAULT '', + result TEXT NOT NULL DEFAULT '', -- one line, at most 200 runes, written once at end + files_total INTEGER NOT NULL DEFAULT -1, -- -1 is unknown, 0 is none + store_path TEXT NOT NULL, -- where the run's own store lives now + engine_session TEXT NOT NULL DEFAULT '', -- the presence session that drives it + PRIMARY KEY (conversation_id, run_id) +); +CREATE INDEX runs_repo_time ON runs (repo, started_at); +CREATE INDEX runs_project_time ON runs (project, started_at); + +CREATE TABLE run_files ( + conversation_id TEXT NOT NULL, + run_id TEXT NOT NULL, + path TEXT NOT NULL, -- repository-relative + PRIMARY KEY (conversation_id, run_id, path) +); +CREATE INDEX run_files_path ON run_files (path); + +CREATE VIRTUAL TABLE runs_fts USING fts5(title, result, + content='runs', content_rowid='rowid'); +``` + +Why these columns: + +- **The key is `(conversation_id, run_id)`.** A run's root id is the chat's row + number (`strconv.FormatUint(row, 10)` in `task_run_continue.go`), so it + repeats across chats. The pair is unique. It is also the key the single store + will use later: the conversation is the parent, the run root is its child. + See [How the index grows into the single store](#how-the-index-grows-into-the-single-store). +- **`repo` beside `project`.** The measurement found that 17 of 26 overlapping + pairs sat in different project folders but the same repository. Overlap is + judged on `repo`. `project` stays for the existing same-folder readers. +- **`files_total` is -1 for unknown.** Today the worker harness records no file + list at all (0 of 138 run tasks measured). An unknown list and an empty list + are two different facts, so they are two different values. A reader never + treats -1 as "touched nothing". +- **`run_files` is capped at 64 paths per run** (the preflight's own + `preflightScanLimit`), repository-relative, with the honest total in + `files_total`. Hot files (`hotFiles`) are stored but never count as overlap. + +### Who writes, and when + +Only the engine, at two moments, each one short transaction. + +1. **Run start.** When a run's store is opened for a new run + (`openBeltRunStore`, `OpenRunPlan`) or a node family root is admitted: one + `INSERT` with `status='running'`, the title, the project, the ground, the + repo and `store_path`. +2. **Run end.** Where the run is already closed today (`Supervisor.Run` ending + in `CompleteRoot`, `EndRoot` on stop, `setAsideRunStore` ending a run as + interrupted, `reportTaskNode` for a node family root): one `UPDATE` of + status, `ended_at`, `result` and the file list, and `store_path` if the + store was set aside. + +The file list at end comes from the run's working copy, not from the model: +the paths changed between the run's base and its landed branch. The engine +already computes the landing (`engine.Land` in `landBeltRun`), so this is one +more reading of a diff it has. For the older engine the node's own `wrote` +list is used. + +The result is the run's own report (the root's `result`), cut to one line with +the existing `taskOutcome` rule. Nothing is summarized at read time. + +### Crash safety + +- **Each write is one transaction** on a WAL database with a busy timeout, the + same settings plandb uses (`persist.go`: `_txlock=immediate`, + `busy_timeout(5000)`, WAL). +- **A failed index write never fails the run.** It is recorded instead, as a + context entry on the run's own root in the run's own store + (`AddContext(root, "brain", ...)`). The run's page can then say "not in the + index: ", which is distinct from a run that is simply not over. +- **A row left `running` by a dead engine** is judged by the rule world.go + already uses: a claim of running is believed only when a fresh presence file + from `engine_session` names that run. Otherwise the reader shows it as ended + without an ending, the same word `setAsideRunStore` writes (`interrupted`). + The pid is never a liveness test (`taskpresence.go`). +- **The store is the truth, the index is a cache of it.** Any row can be + rebuilt from `store_path`. When the engine next opens a store whose root is + terminal but whose index row still says running, it closes the row from the + store. + +### Size bounds + +Measured rate: 481 tasks in 9 days across 87 chats, of which about 260 were +top-level work items (families and runs). That is about 30 index rows a day. + +- A row without files is about 400 bytes. 64 file rows are about 5 KB at worst. +- At 30 rows a day, a year is about 11,000 rows: 5 MB typical, under 60 MB if + every run touched 64 files. +- No retention is needed at that size. If a file grows past 64 MB the engine + drops `run_files` rows older than 180 days and keeps the `runs` rows, so + search still finds the work and only the overlap signal ages out. + +### Migration + +The engine builds the index once, in the background, when `brain.db` is +missing or its schema is unknown. It reads, never writes, the older sources. + +| source | what becomes a row | notes | +| --- | --- | --- | +| `/plandb.db` and every `plandb.db.N` | one row per store with a root: title, status, created and completed times, root result | 16 store files on the measured machine, 11 with rows | +| `/tasks.json` | one row per family root, with files from `wrote` and `changed` | 343 tasks on the measured machine | +| `/tasks.jsonl` | rows for landed work whose chat folder is gone | deduplicated on `(sessionId, id)` against `tasks.json` | +| `~/.codeaf/graph.db` | nothing | measured: its `nodes` table holds one node, the spine root, with no session. Its sessions carry titles only, and 53 of its 54 sessions also have a chat folder | + +While the backfill runs, a reader that finds `meta.built` empty says the index +is still being built. That is a fourth answer, and it too must not render as +"nothing ran". + +## Layer 2: awareness push, through the existing `` block + +The block keeps its grammar: the same tag, the same "Facts, not requests." +line, the same two headings, the same `·`-joined clauses with empty clauses +dropped, no clock in it, the same caps (`deltaLandedRows = 6`, +`deltaLiveRows = 6`). + +What changes is what feeds each half. + +- **The past half** reads the index instead of only the project's + `tasks.jsonl`: landed runs in **other chats**, in this repo or any other, + since `told.json`, first reach 24 hours as today. +- **The present half** keeps reading presence files for open windows. The index + adds nothing to "now", because presence is the fresher source and the liveness + rule lives there. + +A row from another project folder carries one more clause, the project's name +from `projectName`, and only when it differs from this chat's: + +``` + +Work on this project from outside this conversation. Facts, not requests. +recently landed in other windows: +- Fix the nil-map crash · done · internal/reconciler/state.go + Added the guard and the regression test; the parser suite passes. +- Sweep the call sites · done · in home · internal/session/agent.go +running in another window now: +- Survey the config loaders · window "docs pass" · internal/config/load.go + +``` + +### Ranking + +Rows are ranked by overlap with this chat, then cut to the cap. + +| signal | weight | source | +| --- | --- | --- | +| shares a non-hot file with this chat's own runs or its current brief | 4 per file, at most 12 | `run_files` against this chat's rows and `briefFiles` | +| same repository | 3 | `repo` | +| same project folder | 1 | `project` | +| similar title (content-word Jaccard at least 0.5) | 1 | `title` | + +A row with score 0 is never shown. That keeps the block quiet: work in an +unrelated repository is not news to this chat. Ties go to the newest. + +The measurement explains the weights. Files found 26 overlapping pairs. Titles +found almost none: at a strict threshold only 2 pairs across 103,145 +compared. A title is a hint, never a reason on its own. + +### Failure + +When the index cannot be read, the block falls back to what it reads today +(the project `tasks.jsonl` and presence) and adds one line under the lead: +`the index of other chats could not be read (); this is this project's +open windows and its own history only`. When the block would otherwise be +empty, that line is the whole block. A reader can always tell "nothing +elsewhere" from "could not look". + +## Layer 3: awareness pull, through the existing `tasks` tool + +No new tool. `tasks` already takes `query` and `scope`. + +- `scope: "project"` (the default) is unchanged. +- `scope: "everywhere"` today lists live work in other projects. With the index + it also searches **finished** work in every chat and project through + `runs_fts`, ranked by the same score as the push, then by the search score + `SearchTaskIndex` already uses. Rows keep the existing row grammar and are + grouped by project as `taskEverywhereText` groups them. +- A row from another chat carries no id this chat can act on, as today + ("unreachable from here"). It names the chat and its result, so the model can + read it and tell the person, and nothing more. + +The tool description gains no words. The `scope` field's own description +changes from "also lists live work in every other project" to "also searches +every other chat and project, live and finished". That is one field's text, +billed once per request as today. + +Failure: `No task matches "" in this project. Other chats could not be +searched: .` is a different sentence from `No task matches "" in +any chat.` + +## Layer 4: the planning check + +Before a `/task` plans, the engine asks the index one question: which runs in +other chats touched, or are touching, the files this work names, in the same +repository, in the last 24 hours? + +- The files come from `briefFiles` (paths the brief spells), as the preflight + does today, plus this chat's own recent `run_files`. +- The answer goes to the planner as facts in its brief, in the preflight's + existing words: `another window is already in · `, with + `(ended)` or `(running)` added from the row's status. +- **The planner then does one of three things, and says which:** + - **waits**: the other run is running on the same files. The plan starts after + it lands, as `depends_on` already does inside one chat. The wait is + read-only: this run watches the other row's status, it never touches the + other run. + - **joins**: the other run already did the work. The plan reads its result + (the row's result and `store_path`) and plans only what is left. + - **narrows**: the plan drops the shared files from its scope and says so. +- **It never writes into another run.** It does not add a task to it, note it, + steer it or stop it. If the person wants the other run changed, they go to + that chat. + +The preflight's law still holds: this is a fact, not a gate. The person may +start the run anyway, and the planner's choice is shown on the proposal card +so the person can overrule it. + +Failure: `overlap not checked: the index could not be read (<reason>)` goes on +the card. A card that says nothing looks like "no overlap", and those are two +different facts. + +## Later: cross-chat links (read-only) + +A run may record a typed link to a run in another chat. + +- `waits on`: this run starts, or wakes, when that run lands. +- `reads result of`: this run's brief is given that run's result line and + `store_path` to read. + +Links live in the index (`run_links(from_conv, from_run, to_conv, to_run, +kind)`), written by the engine when the planner chose wait or join. They are +read-only by construction: the only thing a link lets the linking run do is +read. Nothing adds tasks to another chat. + +## Later: one scheduler and budget across runs + +Today each run has its own slots and its own dollar limit (`CostUSD` on the +run spec). Later, one scheduler could hold a machine-wide slot count and +budget, reading `running` rows from the index. It needs the index to be +trusted as live, which is why it waits for the liveness rule above to be +proven in use. + +## Measured: how often chats overlap today + +**Data.** Every chat folder on the owner's machine: `meta.json`, `tasks.json`, +every `plandb.db*`, task trajectories, each project's `tasks.jsonl`, and +`graph.db`. Copied as metadata only and analysed off the machine. Only +aggregate numbers are reported here. + +**Span.** Task start times from 2026-09-15 to 2026-09-24, 9 days. + +### Chats and stores + +| measure | value | +| --- | --- | +| chat folders | 87, in 7 project folders (40, 28, 14, 2, 1, 1, 1) | +| chats with a plan store | 9 | +| plan store files, including set-aside copies | 16 (11 with rows) | +| project task index rows | 475, in 4 project folders | + +### Tasks per chat + +| measure | value | +| --- | --- | +| tasks | 481: 343 on the older engine, 138 on the worker harness in 11 runs | +| chats with any task | 34 of 87 | +| tasks per chat, over chats with any | mean 14.1, median 5, 90th percentile 36, max 129 | +| top-level work items per chat, over chats with any | mean 7.7, median 3, max 51 | +| runs per chat with a store | one chat 3, eight chats 1 | + +### Runs in different chats that touched the same file within 24 hours + +A pair is two top-level work items in different chats that share at least one +non-hot file, where the later one started within 24 hours of the earlier one +ending. + +| key | pairs | chat pairs | both live | after the earlier had ended | +| --- | --- | --- | --- | --- | +| repository-relative path | 26 | 4 | 1 | 25 | +| same folder and path | 25 | 3 | 0 | 25 | + +What today's `<elsewhere>` could show of the 26: + +| case | pairs | +| --- | --- | +| both live, same project folder: the present half can show it | 1 | +| after close, same project folder, with an index row: the past half can show it, within its 6-row cap | 8 | +| after close, **different project folder**, same repository: not shown | 17 | + +Shared files per pair: median 1, max 4. + +### Similar titles across chats + +| threshold (content-word Jaccard, or character ratio) | pairs | chat pairs | started within 24 h | in different projects | +| --- | --- | --- | --- | --- | +| 0.6 or 0.85 | 2 | 1 | 0 | 0 | +| 0.5 or 0.75 | 5 | 3 | 2 | 1 | +| 0.4 or 0.65 | 9 | 5 | 2 | 5 | + +103,145 cross-chat title pairs were compared. The highest similarity was 0.75. +Read by hand, the top pairs were a repeated request and several review tasks of +the same kind; the rest shared a verb and not an object. + +### What the numbers say + +1. **Overlap is sequential, not simultaneous.** 25 of 26 pairs happened after + the earlier work had ended. Presence alone can never see these. +2. **Most overlap crosses project folders.** 17 of 26 pairs worked on the same + repository from chats filed under different project folders. Today's + block cannot see any of them. This is why the index keys overlap on `repo`. +3. **Files beat titles.** File overlap found 26 pairs. Titles found 2 at a + strict threshold. Ranking leads with files. +4. **These are lower bounds.** The worker harness records no files (0 of 138 + run tasks carry a list), and 33 tasks carry no start time. Every overlap + involving a harness run is invisible to this measurement, and to codeaf. + Writing the run's changed files at run end is the first thing the index + adds. +5. **The older home store has no task history.** `graph.db` holds one node, the + spine root, with no session. Migration from it is nothing. + +## Later: one universal plandb + +The owner wants, later, one plan store for everything: every conversation a +node under one main root, every run a subtree of its conversation. This section +is the code map of every place that assumes one store is one run, and what +each must become. Counts are verified on `bdb08cfa1`. + +### Counts + +- `RootID()` is called **43 times in 10 non-test files**: `internal/plandb/cli.go` + 18, `internal/run/run.go` 7, `internal/session/task_run_belt.go` 5, + `internal/session/plandb_steer.go` 4, `internal/run/bashworker.go` 2, + `internal/session/plandb_plan.go` 2, `internal/session/plandb_tasks.go` 2, + `internal/plandb/store.go` 1, `internal/session/bashbelt_worker.go` 1, + `internal/session/task_run_continue.go` 1. +- `internal/plandb` is imported by **11 other packages** (7 in non-test code: + `bench/bashloop`, `cmd/codeaf`, `cmd/codeaf-demo-home`, `cmd/plandb`, + `internal/router`, `internal/run`, `internal/session`; and 4 more in tests + only: `internal/enginehost`, `internal/guard`, `internal/remote`, + `internal/tui3`). The earlier estimate of 22 was high. + +Every `RootID()` call must become "this run's root", passed in or held by a +run handle, never read off the store. + +### The map + +| place | what it assumes today | what it must become | +| --- | --- | --- | +| `internal/plandb/store.go` `Open` ("THE ROOT IS THE RUN") and `loadOrCreate` | a store has one project and one root; a different root is "plan store belongs to a different run" | `Open` opens the one store; a new `Run(conv, root)` handle creates or adopts the run's subtree under its conversation node. The refusal moves to the handle: a run handle refuses a root that is not in its conversation | +| `store.go` `RootID`, `Project` | one answer per store | methods on the run handle | +| `store.go` `ReadyLeaves`, `ReadySet`, `Tasks`, `Summary`, `Changed`, `StaleClaims`, `TouchClaims` | the whole store is the run | scoped to the handle's subtree, with an index on `(conversation, root)` so a read is the subtree, not a scan | +| `store.go` `CanFinalize`, `CompleteRoot`, `StopRoot`, `EndRoot`, `closeRoot` | close "the" root | close the handle's root; never walk past it | +| `store.go` `Archive(olderThan)` | archives across the whole file | archives within one subtree, or one conversation | +| `store.go` `transact` and `persist.go` `saveState` | **every write loads the whole state and rewrites every row of `meta`, `tasks`, `deps`, `notes`, `contexts`** (`DELETE FROM` each table, then insert all) | row-level writes before any merge. In one store the current write path would cost every write the size of every conversation's plan, under the one write lock | +| `internal/plandb/cli.go` `cliStore`, `RunEnv` (`PLANDB_RUN`) | the store at `PLANDB_DB` is refused whole unless its root equals `PLANDB_RUN` | the worker opens a run handle for `(PLANDB_CONV, PLANDB_RUN)`; every write is refused unless its target is inside that subtree. The binding gets stronger, not weaker | +| `cli.go` `cliFindStore`, `cliInit` | walks up for `plandb.db`; init refuses an existing store | the store path is fixed; `init` creates a run subtree, and refuses one that exists | +| `internal/session/plandb_plan.go` `planPath`, `PlanStorePath`, `OpenRunPlan` | a path per chat (`<chat>/plandb.db`) or per working copy (`<dir>/.codeaf/plandb.db`); a new run sets the old store aside first | one path; a new run is a new subtree under the chat's node; nothing is set aside | +| `internal/session/task_run_belt.go` `openBeltRunStore`, `setAsideRunStore`, `planArchivePaths` | one live run per path; a new run renames the old file to `plandb.db.N`; the reading verbs find old runs by file suffix | the chat's runs are the children of its node; "set aside" is ending the old subtree as interrupted; `planArchivePaths` becomes a query for the chat's closed runs | +| `internal/run` `Supervisor.Run`, `pass` (`ReadySet`), `absorb`, `rootAwaitingWake`, `treeTerminal`, `launchWakes`, `rootCancelled`, `Start`, the end on `EndRoot` and `CompleteRoot` | the store's tasks are this run's tasks; `treeTerminal` and `launchWakes` loop over `store.Tasks()` | the supervisor holds a run handle; every loop is over the subtree | +| `internal/session` readers: `plandb_steer.go`, `plandb_tasks.go`, `bashbelt_worker.go`, `task_run_continue.go` | read the root off the store they were given | take the run handle | +| `~/.codeaf/graph.db` (`internal/store`) | already one store for the machine, with a unique spine root (`nodes_one_spine_root`) and `session_id` on each node | the precedent, not the target: its `nodes` are the older engine's and hold no task history on the measured machine. Its `sessions` can seed the conversation nodes' titles | + +### How the index grows into the single store + +The index is built so none of it is thrown away. + +- **Same ids.** An index row's key is `(conversation_id, run_id)`. In the single + store the conversation is a node with id `conversation_id` under the main + root, and the run is its child with id `run_id`. A task inside a run keeps + its store id, qualified by the same pair. +- **The conversation is the parent key** in both. Every overlap and ranking + query written against `runs` becomes a query against the conversation nodes' + children, unchanged in shape. +- **`store_path` goes away** when every run lives in one file. Until then it is + how a reader gets from a row to the run. +- **The engine is still the only writer of the conversation level.** In the + single store, a worker's handle is its run's subtree, so the guardrail that + only the engine writes the index becomes: only the engine writes above a + run's root. + +### Risks + +- **One writer at a time, across everything.** SQLite allows one writer per + file. Today each run writes its own file, so two runs never wait on each + other. In one store, every window, every engine and every worker queues on + one lock. With the current whole-state rewrite, the time each write holds + that lock grows with the total size of every plan on the machine. The first + step of the refactor must be row-level writes, measured under the same load + as the BELT DOE. +- **Blast radius.** One corrupt file today loses one run. In one store it loses + every conversation's plans. The single store needs a backup before + migration, a startup integrity check, and a way to open read-only when the + check fails. +- **The performance lead over other CLIs must not regress.** Startup and idle + cost are where codeaf leads today. The single store must not add a scan at + startup, a write on idle, or a lock taken by a window that only reads. The + acceptance for the refactor is the existing performance benchmark, run + before and after on the same machine, with no regression in startup, idle + CPU, or per-step write latency at the measured scale (about 500 tasks) and + at ten times that. +- **The index alone carries none of these risks.** It is a separate file, + written twice per run, so it cannot slow a run's own writes, and losing it + loses nothing a rebuild from the stores cannot restore. + +## Acceptance for the implementing PRs + +Each layer lands on its own, in order, with end-to-end tests on the real +binary: + +1. **Index.** Two chats in different project folders on one repository; each + runs a task touching one shared file. `brain.db` has two rows with the + shared path. Kill the engine mid-run: the row reads as interrupted. Make the + index unreadable: the run still lands, and its page says it is not in the + index. +2. **Push.** The second chat's next turn carries a `<elsewhere>` row for the + first chat's landed work with the project clause. An unreadable index gives + the one fallback line, never an empty block. +3. **Pull.** `tasks` with `scope: "everywhere"` finds the first chat's finished + work by a word in its result. +4. **Planning check.** A `/task` naming the shared file shows the overlap on + the card and the planner's choice (wait, join or narrow). The other run's + store is byte-identical before and after. From d21c36a00e842620c5262e8e3cf777d6a80f3976 Mon Sep 17 00:00:00 2001 From: santoshkumarradha <instrument.santosh@gmail.com> Date: Thu, 24 Sep 2026 12:42:42 -0400 Subject: [PATCH 2/2] docs(changes): entry for the brain index design --- docs/changes/unreleased/1437-brain-index-design.md | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 docs/changes/unreleased/1437-brain-index-design.md diff --git a/docs/changes/unreleased/1437-brain-index-design.md b/docs/changes/unreleased/1437-brain-index-design.md new file mode 100644 index 000000000..01fd732ee --- /dev/null +++ b/docs/changes/unreleased/1437-brain-index-design.md @@ -0,0 +1,11 @@ +--- +kind: internal +title: the design and overlap measurement for a brain index of every run across chats +pr: 1437 +surface: [docs] +--- +One document, no code. `docs/design/brain/DESIGN.md` designs one index of +every run, written only by the engine at run start and end. The index feeds +the existing `<elsewhere>` block, the `tasks` tool and a planning check. The +document measures how often work overlaps across chats today, and maps every +place that assumes one plan store is one run, for a later single store.