fix: rebuild index when mode adds capabilities

[astandrik]

Summary

• persist the effective index capability level in the Project node as index_mode
• force a full rebuild when the requested mode requires capabilities missing from the stored index
• preserve additive data on downgrades, including changed files, by separating requested discovery mode from stored effective extraction mode
• migrate legacy, malformed, or non-exact mode metadata with one conservative rebuild

Root cause

Incremental routing originally considered stored file hashes and the discovered-file threshold, but not the capability level of the stored index. This caused unchanged fast → moderate upgrades to take the incremental no-op path.

After the first fix, changed-file downgrades still re-extracted with the weaker requested mode. That could remove stored capabilities such as SIMILAR_TO edges or full-mode Macro nodes.

The metadata parser also compared yyjson strings with strcmp(). A valid JSON string containing "full" + U+0000 + "garbage" therefore matched "full" at the embedded NUL and could incorrectly select incremental/no-op instead of the conservative rebuild route.

Mode transitions

Stored	Requested	Route / effective behavior
fast	moderate / full	full rebuild
moderate	full	full rebuild
same mode	same mode	existing incremental/no-op path
more capable	less capable	requested mode controls discovery; stored mode is the effective mode for changed-file extraction and capability post-passes
missing / unknown / malformed / non-exact	any	one conservative full rebuild

Downgrades keep the stored index_mode; p->mode and Project metadata are not rewritten. Files outside requested discovery continue to use the existing mode-skipped/hash mechanism.

The public MCP API and SQLite schema are unchanged. Existing changed-file threshold and ADR preservation behavior are unchanged.

Regression coverage

• fail-before: unchanged fast → moderate left SIMILAR_TO at 0 because routing selected incremental no-op
• pass-after: the same transition rebuilds, creates deterministic similarity data, and records index_mode=moderate
• fail-before: after a moderate build, changing a Go file and requesting fast removed its similarity contribution
• pass-after: changed-file moderate → fast keeps the same SIMILAR_TO count and index_mode=moderate; a following moderate no-op remains stable
• fail-before: after a full build, changing a C file and requesting moderate removed full-only Macro nodes
• pass-after: changed-file full → moderate preserves Macro nodes and index_mode=full; a following full no-op remains stable
• legacy Project metadata {} triggers a rebuild and restores index_mode
• fail-before: an index_mode value containing U+0000 after full was treated as full, selected incremental no-op, and retained a DB-only sentinel
• pass-after: exact length-aware yyjson comparison rejects the NUL-suffixed value, rebuilds once, removes the sentinel, and restores index_mode=fast

Verification

• fail-before downgrade suite: 222 passed, 2 failed
• fail-before embedded-NUL regression: 224 passed, 1 failed
• ASAN_OPTIONS=detect_leaks=0 build/c/test-runner pipeline — 225 passed
• scripts/test.sh — 6015 passed, 1 existing platform skip; watchdog/security checks and production build passed
• make -f Makefile.cbm cbm — passed
• make -f Makefile.cbm lint-format lint-no-suppress — passed
• git diff --check — passed

scripts/lint.sh was attempted locally. The environment has no cppcheck, and its Homebrew clang-tidy reports broad pre-existing errors outside this diff. The independent format/no-suppression gates passed; the GitHub lint job is the authoritative full lint check.

[DeusData]

Thanks for the mode-capability fix and regression coverage. This is triaged as a normal-priority 0.9.1-rc correctness PR. Review will focus on the capability ordering, rebuilding only when a requested mode adds unavailable graph data, preserving valid incremental state, and keeping same-mode reindex behavior unchanged.

[astandrik]

Thanks for the mode-capability fix and regression coverage. This is triaged as a normal-priority 0.9.1-rc correctness PR. Review will focus on the capability ordering, rebuilding only when a requested mode adds unavailable graph data, preserving valid incremental state, and keeping same-mode reindex behavior unchanged.

@DeusData I ran into this while working with a really large repository: https://github.com/ydb-platform/ydb

Version 8 wasn’t able to index it at all, but version 9 managed to handle it successfully. Thanks for the improvements — I really love your codebase memory tool!

[Copilot]

Pull request overview

This PR fixes incremental routing so it accounts for the capability level of the stored index (via Project.index_mode), forcing a full rebuild when a requested mode requires capabilities the existing index does not have, while preserving stronger stored capabilities during requested-mode downgrades.

Changes:

• Persist index_mode on the Project node during structure pass and parse it during routing to detect capability upgrades.
• Route fast → moderate/full (and moderate → full) transitions to a full rebuild instead of an incremental no-op.
• Run incremental changed-file extraction using an explicit effective_mode (stored mode) to preserve additive capabilities across downgrades, and add regression tests for mode transitions and malformed/NUL-suffixed metadata.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
tests/test_pipeline.c	Adds regression tests covering mode upgrade rebuilds, downgrade preservation behavior, and malformed/NUL-suffixed index_mode metadata handling.
src/pipeline/pipeline.c	Persists Project.index_mode, parses stored mode, and blocks incremental routing when requested mode needs missing capabilities.
src/pipeline/pipeline_internal.h	Updates incremental pipeline contract to accept an effective_mode for changed-file extraction.
src/pipeline/pipeline_incremental.c	Applies effective_mode to incremental extraction (incl. macro gating) while keeping requested mode for discovery/exclusions.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.