Skip to content

docs(design): the brain index, shared reads and separate writes - #1437

Closed
santoshkumarradha wants to merge 2 commits into
santos/dev2from
design/brain-index
Closed

santoshkumarradha wants to merge 2 commits into
santos/dev2from
design/brain-index

Conversation

@santoshkumarradha

Copy link
Copy Markdown
Member

Design only. No product code.

docs/design/brain/DESIGN.md designs one small index of every run across chats and projects: ~/.codeaf/v3/brain.db, one row per run, written only by the engine at run start and run end. Each run keeps its own plan store for writes, so the PLANDB_RUN isolation from #1417 stays.

Four layers

  1. Index. Schema keyed (conversation_id, run_id). The engine is the only writer. It covers crash safety, size bounds, and a read-only backfill from the plan stores, tasks.json, tasks.jsonl and graph.db.
  2. Push. The existing <elsewhere> block reads the index too. It keeps its grammar and caps. Rows are ranked by shared files first, then repo, project and title.
  3. Pull. The existing tasks tool with scope: "everywhere" also searches finished work in every chat. No new tool.
  4. Planning check. Before a /task plans, it checks the index for overlap and then waits, joins or narrows. It never writes into another run.

Later: read-only cross-chat links, one scheduler and budget across runs, and a code map for one universal plandb, with verified counts and risks.

Measurement (aggregates only)

Every chat folder on one working machine over 9 days: 87 chats in 7 project folders, 481 tasks.

  • Tasks per chat, over the 34 chats that had any: median 5, mean 14.1, max 129.
  • Work items in different chats that touched the same file within 24 h: 26 pairs (4 chat pairs).
    • Both live at once: 1. After the earlier one ended: 25.
    • Today's <elsewhere> could show 9 of them. The other 17 were on the same repository from chats filed under a different project folder.
  • Similar titles across chats: 2 pairs at a strict threshold, out of 103,145 compared. Files are the signal. Titles are not.
  • Lower bound: worker-harness tasks record no file list (0 of 138).

Findings from the code, beyond the brief

  • The past half of <elsewhere> already sees closed chats, but only in the same project folder, and the project folder is the launch folder, not the repository.
  • Every plandb write reloads the whole store and rewrites every row (transact then saveState). That is fine per run, but it has to become row-level writes before any single store.
  • RootID() is called 43 times in 10 non-test files, as estimated. internal/plandb is imported by 11 other packages (7 in non-test code), not 22.

Change entry: docs/changes/unreleased/<this PR>-brain-index-design.md.

A design for one index of every run across chats and projects, written
only by the engine at run start and end, feeding the existing <elsewhere>
block, the tasks tool and a planning check. Includes the overlap
measurement and the code map for a later single plan store.
@santoshkumarradha

Copy link
Copy Markdown
Member Author

Combined into #1443, which now carries this design doc and its change entry together with the first steps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant