fix(search): knowledge_search rebuilds the BM25 index on every call — 330 ms uncached against 3 to 7 ms cached at 4,728 pages

[drewstone]

Version checked: agent-knowledge 15.0.2 (tag v15.0.2, 1b1408e), the Discovery Lab's installed pin. Every file cited below is byte-identical on origin/main (a7fb5fc, 15.0.3), so the line numbers hold on both.

What the code does

• src/knowledge-tools.ts:87-92 — the knowledge_search tool calls stores.loadChain(runId) and then buildKnowledgeBrief(chain, input.question, ...) on every call.
• src/run-scoped.ts:173-192 — loadChain re-reads every page file of the run store, each ancestor, and the shared root through loadKnowledgePages on every call, and returns fresh page objects.
• src/knowledge-brief.ts:93-102 — buildKnowledgeBrief calls searchKnowledgePages(pages, question, { limit, excludeInvalidated, tags, kinds }) and passes no lexicalIndex.
• src/search.ts:89-91 — with no lexicalIndex, searchKnowledgePages runs buildKnowledgeLexicalIndex(pages), a full BM25 pass over every page, for that one query and then discards it.
• src/search.ts:155-164 — assertLexicalIndexMatches accepts a supplied index only when lexicalIndex.pages[ordinal] === pages[ordinal] by object identity, so an index built over one loadChain result can never be reused for the next.

Only FileSystemSearchProvider (src/filesystem-search-provider.ts:62-66) keeps the lexical index between searches, as the closing comment on #132 records. Every other caller builds it per query: knowledge_search above, src/cli.ts:188, src/inspect.ts:104 and :111, src/retrieval-eval.ts:322, src/eval-readiness.ts:140, and src/verified-research-loop.ts:517. The Lab's experiment and operator-audit scripts (experiments/knowledge-continuation-v1/method.mjs, artifacts/company-frontiers-decisive-20260909-r7/operator-audits/knowledge-composition-check.mjs) mount createKnowledgeTools, not the provider.

Measured

From the findings file (Discovery #159, row 8b), measured on the Lab kb store on 2026-09-09:

• 4,728 pages in index.json (21.7 MB), 5,040 Markdown files.
• Parse index.json: 83 ms. Lexical build: 335 ms. Uncached search: 328-338 ms. Cached search (lexical index supplied): 3-7 ms. RSS 499 MB after 3 queries.

Re-checked read-only while filing: kb/.agent-knowledge/index.json is 21,751,164 bytes with 4,728 pages (generatedAt 2026-09-09T03:12:59.829Z). The store grows by about 50-100 pages per day (#132) and the per-query cost is linear in pages, so every knowledge_search a director or lead makes pays a full-corpus index build, and a months-long run repeats it thousands of times (projection; row Y7 of the findings).

knowledge_search also caps limit at 50 (src/knowledge-tools.ts:58). That is not a defect; noted because the findings row lists it.

Correct behavior

One lexical index per store state, shared across searches on the tool path. Two acceptable shapes:

1. Persist the lexical index (or enough to rebuild it incrementally) alongside index.json under .agent-knowledge/, invalidated by the same mutation lock that guards putIndex.
2. Keep an in-memory cache keyed on the page set's content identity (page ids plus their digests or updatedAt, or the index generatedAt), so assertLexicalIndexMatches no longer requires object identity and successive loadChain results can reuse it.

Either way, knowledge_search on an unchanged 4,728-page store should cost the cached figure (3-7 ms), not the uncached one (328-338 ms), and loadChain should not re-read 5,040 files when nothing changed.

Findings: https://github.com/tangle-network/discovery/blob/4cbc5406707cdf753e3b950ad0e38cb67914da95/docs/research/159-long-horizon-ceilings.md (row 8b and "Upstream gaps"). Filed for Discovery map tangle-network/discovery#136 through tangle-network/discovery#167.

🤖 Generated with Claude Code