feat(sessions): search a project's earlier agent sessions — codegraph sessions + codegraph_sessions

[bompus]

A code graph answers "how does X work"; it cannot answer "why is X like this" or "what did the last session decide about Y". That history lives in the transcripts the agent already wrote — for one active project here, 237 JSONL files and 238 MB that nobody greps. This adds a second tool over that corpus.

What it does. codegraph_sessions (MCP) and codegraph sessions <words> (CLI) run a full-text search over the prose of a project's Claude Code sessions: user prompts, assistant text and compaction summaries. Tool calls, tool results, thinking blocks, meta entries and anything under 20 characters stay out. FTS5 with porter stemming and BM25 rank, so trimmed finds trimming and the passage that says most about the words comes first. A hit names its session id, title, role, timestamp and the matching passage with the words marked; role, sinceDays, session (an id prefix) and any (OR the words) narrow or widen. The output is text an agent can act on: the session id is what claude --resume and the transcript file take.

Where the index lives. .codegraph/sessions.db, its own file beside codegraph.db, through the same createDatabase adapter. Kept separate on purpose: putting the tables in the graph database touches schema.sql, migrations.ts and the bulk-load FTS rebuild for no query gain, and the graph's nodes_fts uses unicode61 without stemming, which is right for identifiers and wrong for prose. The index refreshes on each call for files whose size or mtime moved and forgets files that are gone. Measured on the 237-file project: 6.5 s for the first index on the bundled Node, ~140 ms for a query afterwards with one live session changed.

First-run concurrency. Two commits in this branch (5397d49, 304664d, +89/-1) are not about search — they close a race on the brand-new sessions.db, which is exactly the file parallel tool calls all open at once. busy_timeout is set before the schema and version writes, and the PRAGMA journal_mode = WAL conversion retries on either transient error within that same budget. Attributing failures by frame put every one of them on the pragma, never the schema write: six concurrent converters on a fresh file failed 22/300 without the retry and 0/480 with it, arms round-robined. It is first-run only, since WAL is persistent and nothing converts the file twice. Two unit tests pin the retry contract and the rethrow of a non-transient error; the evidence that the race is closed is the harness, not the suite. Full measurement, including the busy_timeout reasoning I had to correct along the way: #1702 (comment)

Surface. The tool joins codegraph_explore in DEFAULT_MCP_TOOLS and in the tiny-repo core set. It is a different question over a different corpus, so it cannot steer a mis-pick against explore, and the measured argument for one tool — fewer mis-picks between look-alike code tools — does not reach it. The server instructions gain one routing bullet ("why is this like this / what did the last session decide" → sessions) and say "for code there is one tool" instead of "a single tool". "sessions": false in codegraph.json opts a project out (the tool then answers with the switch's name); CODEGRAPH_SESSIONS_DIR points at another transcript directory, which the tests use. The Claude Code slug is the project path with non-alphanumerics as -; on Windows the drive letter is lowercased in practice, so the lowercased slug is tried second.

Shape. src/sessions/claude-code.ts is the reader (one module per agent host, so a Cursor or Copilot reader is a second file with the same SessionDoc output); src/sessions/index.ts is the index and query (SessionsIndex, querySessions, formatSessionHits). Every query word is quoted before it reaches FTS5, so flags, paths and punctuation cannot break the match syntax. Nothing in the graph's schema, extraction or explore path changes.

Tests. __tests__/sessions-index.test.ts (doc extraction, quoting, stemming, filters, incremental refresh, a replaced and a deleted file) and __tests__/cli-sessions-command.test.ts (end to end against the built CLI with CODEGRAPH_SESSIONS_DIR); mcp-tool-allowlist updated for the two-tool default; mcp-tool-annotations and mcp-unindexed pass unchanged. tsc clean. The full suite on my Windows host shows no new failure against the same host's baseline (the failures there are EPERM temp-dir cleanup and unbuilt viewer assets, the same set with and without this change).

Coverage. Claude Code only, in this PR. Of the nine agents the README supports, the others keep their conversations in stores this reader does not open: Codex CLI, Gemini CLI and the Copilot CLI write per-session JSON or JSONL files under their own home directories (~/.codex/sessions/, ~/.gemini/tmp/<hash>/chats/, ~/.copilot/session-state/), each a reader module of the same shape as claude-code.ts once the project can be matched to a session; Cursor, Antigravity and Kiro are VS Code descendants that keep chat inside the IDE's SQLite state, and OpenCode and Hermes Agent keep theirs in their own SQLite databases. A project with none of these indexed answers with the guidance text, not an error. The SessionDoc contract (ts, role, text per turn, one file per session) is what a second reader has to produce; the FTS index, the CLI and the tool do not change per host.

If the shape is acceptable, the other hosts are follow-on PRs of one reader module each — Codex CLI, Gemini CLI and the Copilot CLI first, since their stores are plain files — and I am glad to take them, or to review someone's who has the store to test against.

Not in this PR: those readers, embeddings, cross-project search, watcher-driven refresh, a viewer tab.

[bompus]

Pushed a second commit (b1ab174) after the first live run of codegraph_sessions from a Claude Code session surfaced two problems:

Parallel calls failed with "database is locked". In daemon mode read tools run on the worker-thread pool, so several codegraph_sessions calls in one response each open their own connection to sessions.db, all notice the same changed transcript, and all try to write it. node:sqlite defaults to a zero busy timeout, so every call after the first errored. The connection now sets busy_timeout (5 s), and a file is re-indexed under BEGIN IMMEDIATE after re-reading its files row, so the threads that lost the race skip the file instead of indexing it twice. The new test holds the write lock from a worker thread for 300 ms and checks both: no throw, and no duplicate docs.

Prompts sent mid-turn were never indexed. Claude Code stores a message the user sends while a turn is running as an attachment entry (attachment.type: "queued_command", text under attachment.prompt), not a user message. The reader now indexes those as the user. PRAGMA user_version records the reader version, so an index built by the earlier reader is re-read once (about 2 s for 239 transcripts on this machine, then back to ~130 ms per call).

Tests: sessions-index.test.ts 8/8 on Node 24.

[bompus]

Third commit (5397d49): busy_timeout is now set before the constructor's CREATE TABLE and user_version writes, which race between worker threads the same way the per-file writes did. Verified from a live Claude Code session: six parallel codegraph_sessions calls in one response, all succeed.

[bompus]

Probed this one properly. The functional half is clean; the concurrency half is much better than before but not finished, and I think I can point at exactly which statement is left.

Method caveat first. I intended to run this against real Claude Code transcripts and was blocked from reading them, so I did not work around it — I generated a corpus instead: 40 transcripts × 25 turns, user + assistant text blocks plus tool_use entries, ISO timestamps, one nested subagent directory. That means the functional numbers below are on synthetic prose, and I have not verified the reader against real-world transcript shapes.

(One thing that cost me an hour and is worth a line in the reader's doc comment: transcriptDocs requires e.timestamp, so an entry without one is silently dropped. My first corpus had no timestamps and indexed 0 docs with no error. That's the correct behaviour, just very quiet.)

Functional results — all correct:

• refresh → {"files":40,"refreshed":40,"docs":2000} in 31 ms.
• Re-refresh → {"files":40,"refreshed":0,"docs":0} in 1 ms. The mtime/size short-circuit works.
• Searching ls -la (which appears only inside tool_use blocks) → 0 hits. Tool traffic genuinely stays out of the index, as the module docstring claims.
• AND-by-default: "restart zzzz" → 0 hits; the same query with any: true → 10 hits.

The concurrency fix: real, large, and incomplete.

I ran 6 concurrent processes each doing SessionsIndex.open(db) + refresh() against the same brand-new database — the case the head commit's message describes.

arm	failures
a6edfe6 (feature commit, before both concurrency fixes)	105 / 120
b1ab174 (after the first fix)	12 / 48
5397d49 (head, this PR)	11 / 120

So it's roughly an order-of-magnitude improvement and the ordering change is clearly right. But the residual is not noise, and it's all one thing.

Every surviving failure is in open(), none in refresh() — I attributed them by frame. Replaying open()'s exact statement order with a label on each step, all 11/120 failures land on one line:

11x FAIL journal_mode=WAL :: database is locked

That is consistent with the docs: the busy handler does not cover this pragma the way it covers ordinary writes, so setting busy_timeout first — correct and necessary as it is — cannot protect it. Since WAL is persistent in the file, this only bites when several processes race a database that doesn't exist yet, which is exactly the first-run case.

I tested two remedies against the head's own order, round-robining the arms across trials so no arm ate the cold start:

arm	failures / 300 runs	error kinds
head (unconditional journal_mode = WAL)	40	database is locked + disk I/O error
read-first (set WAL only if not already wal)	20	database is locked + disk I/O error
retry (re-issue WAL while it reports busy, inside the existing budget)	16	disk I/O error only

The count difference is worth less than the kind difference: the retry arm never once produced database is locked. Read-first helps less than it looks like it should, because on a genuinely fresh database every process reads delete and they all try to convert anyway.

What retry does not fix is a second, distinct failure — disk I/O error, which I believe is -shm creation contention and which no busy-handler strategy will reach. So I'd frame the remaining work as two separate items, not one.

Caveat on the absolute numbers: Windows + WAL, and 6 processes hammering one brand-new DB is deliberately aggressive, so the rates are environment-specific and I wouldn't quote them as "the" failure rate. The arm-to-arm comparison is the part I'd stand behind — same host, same corpus, interleaved arms.

Recommendation: the PR is a clear improvement and I don't think the residual should block it, but the head commit message currently reads as though the race is closed, and it isn't. Either a retry around the WAL pragma, or an honest "first-run concurrent open can still fail" note, would leave the next person in a better spot than the current wording does.

[bompus]

Correcting myself on the mechanism above. I said the busy handler doesn't cover the journal_mode pragma. That's wrong, and I only caught it because the regression test I wrote for it passed against unpatched code — which it had no business doing.

busy_timeout does cover an ordinary lock wait on the conversion. Measured: a holder takes BEGIN EXCLUSIVE on a fresh database and commits after 400 ms; a second connection with a 5 s budget waits 429 ms and converts successfully. So a connection that merely holds the file is waited out, exactly as it is for any other write.

What actually fails is narrower: several processes converting the same brand-new database at the same moment. They collide inside the conversion itself rather than queueing on a lock, and the loser gets one of the two transient errors. Same 5 s budget, 6 concurrent converters on a fresh file: 2/60 still threw. That's the real defect, and it's why the failure only ever appears on first run — WAL is persistent, so nothing converts the file twice.

Everything I reported measurement-wise stands (the 11/120 attribution to this one pragma, the arm comparison); it was my explanation of why that was wrong.

The upside of chasing it: the disk I/O error turns out to be transient too, so the fix is complete rather than partial. Retrying both errors inside the existing budget:

arm	failures
head, unconditional conversion	22 / 300
retry on database is locked only	22 / 300
retry on both	0 / 480 (two independent samples)

I also tested simply tolerating a failed conversion and staying in the default journal mode — that does not work: the connection doesn't survive the failure, and the following statement fails on it too (FAIL mode, FAIL schema). So retry is the option, not fallback.

Fix is up as bompus#1, based on this branch (pr/session-index) rather than main, so it merges into this PR instead of racing it. Windows/WAL caveat from my earlier comment still applies to the absolute rates; the arm comparison is interleaved and same-host.

[bompus]

Folding the WAL-conversion follow-up into this branch, so it lands with the feature rather than trailing it. Preserving the full write-up here, because the PR that carried it (bompus/codegraph#1) closes on merge and its evidence is the reason for the change.

---

Follow-up to #1702 — targets that branch, not main. It closes the residual first-run race I measured in my comment there.

What's actually broken

#1702's head commit sets busy_timeout before the schema writes, which is right and fixes most of it. But six processes opening the same fresh sessions.db still failed 11 times in 120 runs, and attributing each failure by frame put every single one on one statement — PRAGMA journal_mode = WAL — never the schema write, never refresh().

I first assumed busy_timeout just doesn't apply to that pragma. That was wrong, and I want to be precise about it because it changes the fix. It does apply: a holder takes BEGIN EXCLUSIVE on a fresh database and commits after 400 ms, and a second connection with a 5 s budget waits 429 ms and converts fine. An ordinary lock wait is covered.

What isn't covered is several processes converting the same brand-new file at the same moment. They collide inside the conversion rather than queueing on a lock. Same 5 s budget, six concurrent converters on a fresh file: 2/60 still threw. This is a first-run-only defect — WAL is persistent, so nothing converts the file twice — but the first run is exactly when parallel tool calls all open the index at once.

The fix

Retry the conversion on either transient error inside the budget it already has, with jitter.

Both errors are transient. database is locked is the conversion losing the race; disk I/O error is the -shm file being created underneath a concurrent opener (I saw this on Windows). Retrying only the first is what makes the difference between "mostly fixed" and fixed:

arm	failures
#1702 head, unconditional conversion	22 / 300
retry on database is locked only	22 / 300
retry on both	0 / 480 (two independent samples)

Arms were round-robined across trials so no arm ate the cold start.

I also tested just tolerating a failed conversion and staying in the default journal mode. That does not work — the connection doesn't survive the failure and the next statement on it fails too (FAIL mode, FAIL schema in my harness). So retry is the option, not fallback.

Tests

Two, plus one existing behaviour now pinned:

• enterWalMode retries each transient error and succeeds — driven directly, because the collision only reproduces probabilistically and I'm not shipping you a flaky test.
• enterWalMode rethrows a non-transient error rather than spinning until the deadline.
• A held-lock open converts once the holder commits — this documents the part busy_timeout does cover, which is what I originally got wrong.

Being straight about the limits of these: the unit tests exercise the retry contract, they don't reproduce the race. The evidence that the race is closed is the 480-run harness, not the suite.

enterWalMode is exported for that first test; the alternative was a probabilistic test.

Verification

• tsc --noEmit clean.
• __tests__/sessions-index.test.ts — 11/11 pass, and the two new tests fail without the change.
• The full suite has widespread failures on my host (Windows, no native kernel binary, git-fixture errors). I checked them at the parent commit with the change reverted and they are identical, so they're pre-existing and unrelated — but I haven't run a clean full suite, and you should assume CI is the real check.

Absolute rates are Windows + WAL with six processes hammering one new database, so they're environment-specific. The arm-to-arm comparison is same-host and interleaved, which is the part I'd stand behind.