refactor: Replace progressStore with file-based IndexStatus

[doITmagic]

Description

Replaces the old, complex progressStore indexing progress mechanism with a simple file-based IndexStatus system.

Problem

The previous system had two separate file walks (one for counting, one for indexing), an in-memory progressStore with preRegister/update/carry-over/flusher logic, and multiple intermediate structs (IndexingProgressSummary, LangProgressItem). This made progress reporting confusing, inaccurate, and hard to maintain.

Solution

• Single source of truth: {workspaceRoot}/.ragcode/index_status.json — a simple JSON file written by the indexer during processing
• Direct write from indexer: The Progress callback in pkg/indexer/service.go writes OnDisk, Changed, Processed counts per language directly to the status file
• Direct read by tools: MCP tools read the status file via indexer.LoadIndexStatus() — no intermediate transformations
• Lifecycle: starting → running (first file processed) → completed/failed

What was removed

• progressStore (preRegister, update, carry-over, flusher goroutine)
• IndexingProgressSummary, LangProgressItem, BuildIndexingProgress, formatAge, buildIndexingMessage
• Auto-resume logic from SearchCode/HybridSearchCode (redundant with DetectContext which already triggers re-indexing on first tool call)
• resumeAttempts field from Engine struct

What was added

• pkg/indexer/index_status.go — IndexStatus, LangStatus, SaveIndexStatus, LoadIndexStatus
• pkg/indexer/index_status_test.go — round-trip and missing file tests
• Progress callback wiring in engine.IndexWorkspace

Files changed (22 files, +208 / -1182 lines)

• Moved: engine/index_progress.go → pkg/indexer/index_status.go
• Modified: engine.go, response.go, smart_search.go, smart_search_pipeline.go, index_workspace.go, list_package_exports.go, find_usages.go, call_hierarchy.go, evaluate_ragcode.go, skills.go, read_file_context.go
• Tests updated: engine_searchcode_test.go, engine_*_test.go, health_metrics_test.go, detector_test.go

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Checklist:

• I have performed a self-review of my own code
• I have formatted my code with go fmt ./...
• I have run tests go test ./... and they pass
• I have verified integration with Ollama/Qdrant (if applicable)
• I have updated the documentation accordingly

[Copilot]

Pull request overview

Replaces the in-memory progressStore indexing progress mechanism (with preRegister, flusher goroutine, carry-over logic) with a simpler file-based IndexStatus system in pkg/indexer/index_status.go. MCP tools now read status directly from {workspaceRoot}/.ragcode/index_status.json via indexer.LoadIndexStatus(). This is a significant simplification that removes ~1000 lines of complex concurrency code.

Changes:

• New pkg/indexer/index_status.go with IndexStatus/LangStatus structs and SaveIndexStatus/LoadIndexStatus functions, replacing the old engine/index_progress.go with its progressStore, flusher goroutine, and deep-copy logic
• All MCP tools updated to use the new IndexingStatus field (backed by indexer.IndexStatus) instead of IndexingProgress (backed by IndexingProgressSummary), with the JSON tag preserved as "indexing_progress" for backward compatibility
• Removed auto-resume logic from SearchCode/HybridSearchCode and the resumeAttempts throttle, as workspace re-indexing is now triggered via DetectContext

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
pkg/indexer/index_status.go	New file: IndexStatus/LangStatus types, SaveIndexStatus, LoadIndexStatus
pkg/indexer/index_status_test.go	New file: round-trip and missing-file tests
internal/service/engine/engine.go	Remove progressStore/resumeAttempts, add GetIndexStatus, wire Progress callback with file-based status
internal/service/engine/index_progress.go	Deleted: old progressStore and all related types/functions
internal/service/engine/index_progress_test.go	Deleted: tests for removed progressStore
internal/service/tools/response.go	Replace IndexingProgressSummary with IndexingStatus, rename helper to ContextFromWorkspaceWithStatus
internal/service/tools/smart_search_pipeline.go	Use LoadIndexStatus, replace dynamic fallback note with static string
internal/service/tools/smart_search.go	Simplify indexing error messages, remove progress attachment
internal/service/tools/find_usages.go	Switch to GetIndexStatus/ContextFromWorkspaceWithStatus
internal/service/tools/call_hierarchy.go	Switch to GetIndexStatus/ContextFromWorkspaceWithStatus
internal/service/tools/list_package_exports.go	Switch to ContextFromWorkspaceWithStatus, remove idx-based status override
internal/service/tools/index_workspace.go	Use LoadIndexStatus directly
internal/service/tools/skills.go	Set IndexingStatus: nil
internal/service/tools/evaluate_ragcode.go	Remove unused wctxID/wctxRoot, set IndexingStatus: nil
internal/service/tools/read_file_context.go	Set IndexingStatus: nil
internal/service/tools/tests/health_metrics_test.go	Remove tests for deleted types/functions
internal/service/engine/engine_searchcode_test.go	Remove auto-resume test
internal/service/engine/engine_nonblocking_search_test.go	Remove progress.stop() cleanup
internal/service/engine/engine_fallback_search_test.go	Remove progress.stop() cleanup
internal/service/engine/engine_sticky_test.go	Remove progress.stop() cleanup
pkg/workspace/detector/detector_test.go	Isolate test to avoid picking up .ragcode markers from parent dirs
cmd/rag-code-mcp/main.go	Version bump to 2.1.63

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 9 comments.

Comments suppressed due to low confidence (2)

internal/service/tools/call_hierarchy.go:136

• The check if idx != nil at line 133 is now semantically incorrect. With the old in-memory progressStore, GetIndexProgress returned nil when no indexing was active. With the new file-based system, GetIndexStatus reads from disk and returns non-nil even for a completed previous run (the file persists). This means the response will always be indexing_in_progress when no collections exist but a previous run left an index_status.json file — even if indexing completed hours ago.

This needs to check the actual state (e.g., EndedAt is empty and Error is empty) to determine if indexing is truly in progress. Without a State field on IndexStatus, you'd need something like: if idx != nil && idx.EndedAt == "" && idx.Error == "".

			if idx != nil {
				resp.Status = "indexing_in_progress"
				resp.Data = map[string]any{"indexing": idx}
			}

internal/service/tools/find_usages.go:105

• Same issue as in call_hierarchy.go: if idx != nil at line 103 will now be true even for a completed previous indexing run (the file persists on disk), incorrectly changing the status from indexing_required to indexing_in_progress. This check needs to verify that indexing is actually ongoing (e.g., idx.EndedAt == "").

			if idx != nil {
				resp.Status = "indexing_in_progress"
			}

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 8 comments.

Comments suppressed due to low confidence (2)

internal/service/tools/call_hierarchy.go:136

• Same bug as in find_usages.go: idx is loaded from the persisted index_status.json file. Once any indexing run has completed, the file exists and idx != nil is always true — causing the response status to be incorrectly set to "indexing_in_progress" even when no indexing job is active.

The condition on line 133 should check whether indexing is actually in progress (e.g., via ActiveIndexingJobs() or checking idx.EndedAt == "") rather than just checking file existence.

	idx := t.engine.GetIndexStatus(wctx.Root)

	visited := make(map[string]bool)

	rootNode := &CallNode{Name: symbolName}

	// Try to find root symbol info
	rootRes := t.findSymbolInfo(ctx, wctx.ID, symbolName)
	if rootRes != nil {
		rootNode.Type, _ = rootRes.Point.Payload["type"].(string)
		rootNode.FilePath, _ = rootRes.Point.Payload["file_path"].(string)
		rootNode.Package, _ = rootRes.Point.Payload["package"].(string)
	} else {
		// If nothing is indexed yet, ExactSearchPolyglot will return ErrNoCollectionsFound.
		// Signal indexing status instead of returning an empty hierarchy.
		_, sErr := t.engine.ExactSearchPolyglot(ctx, wctx.ID, map[string]interface{}{"name": symbolName}, 1)
		var noCollections *engine.ErrNoCollectionsFound
		if errors.As(sErr, &noCollections) {
			resp := ToolResponse{
				Status:  "indexing_required",
				Message: fmt.Sprintf("⏳ Workspace '%s' is not indexed yet. Indexing is required for complete call hierarchy results.", wctx.Root),
				Context: ContextFromWorkspaceWithStatus(wctx, t.engine),
			}
			if idx != nil {
				resp.Status = "indexing_in_progress"
				resp.Data = map[string]any{"indexing": idx}
			}

internal/service/tools/find_usages.go:105

• Bug: idx is now loaded from the persisted index_status.json file via GetIndexStatus(). Unlike the old GetIndexProgress() which returned non-nil only when an active in-memory job was running, the file persists after indexing completes. This means idx != nil will always be true once any indexing run has occurred, causing the status to incorrectly change from "indexing_required" to "indexing_in_progress" even when indexing finished long ago.

To fix: either check if an indexing job is currently active (via ActiveIndexingJobs() or indexingJobs.Load(wctx.ID)), or check a specific field on the status (e.g., s.EndedAt == "") before treating it as "in progress".

	idx := t.engine.GetIndexStatus(wctx.Root)
	allResults, err := t.engine.ExactSearchPolyglot(ctx, wctx.ID, filter, 100)
	if err != nil {
		var noCollections *engine.ErrNoCollectionsFound
		if errors.As(err, &noCollections) {
			resp := ToolResponse{
				Status:  "indexing_required",
				Message: fmt.Sprintf("⏳ Workspace '%s' is not indexed yet. Indexing is required for complete results.", wctx.Root),
				Context: ContextFromWorkspaceWithStatus(wctx, t.engine),
			}
			if idx != nil {
				resp.Status = "indexing_in_progress"
			}

[Copilot]

Pull request overview

Copilot reviewed 34 out of 34 changed files in this pull request and generated 9 comments.

[Copilot]

Pull request overview

Copilot reviewed 35 out of 36 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

internal/service/tools/find_usages.go:105

• In the ErrNoCollectionsFound branch, resp.Status is switched to "indexing_in_progress" whenever idx != nil. With the new file-based IndexStatus, idx will be non-nil for any workspace that has ever written index_status.json (even if indexing already completed), so this can misreport "indexing_in_progress". Consider keying this on a real in-progress signal (e.g., idx.EndedAt == "" / idx.Error == "" or checking Engine.ActiveIndexingJobs for wctx.ID) instead of mere file existence.

	idx := t.engine.GetIndexStatus(wctx.Root)
	allResults, err := t.engine.ExactSearchPolyglot(ctx, wctx.ID, filter, 100)
	if err != nil {
		var noCollections *engine.ErrNoCollectionsFound
		if errors.As(err, &noCollections) {
			resp := ToolResponse{
				Status:  "indexing_required",
				Message: fmt.Sprintf("⏳ Workspace '%s' is not indexed yet. Indexing is required for complete results.", wctx.Root),
				Context: ContextFromWorkspaceWithStatus(wctx, t.engine),
			}
			if idx != nil {
				resp.Status = "indexing_in_progress"
			}

internal/service/tools/call_hierarchy.go:136

• The "indexing_in_progress" status is currently set whenever idx != nil, but IndexStatus will be non-nil for any workspace that has an index_status.json from a previous run. This can incorrectly report indexing as in progress when indexing is actually completed (or stale). Consider switching this condition to something that reflects an active run (e.g., idx.EndedAt == "" and idx.Error == "" / checking Engine.ActiveIndexingJobs for wctx.ID).

	idx := t.engine.GetIndexStatus(wctx.Root)

	visited := make(map[string]bool)

	rootNode := &CallNode{Name: symbolName}

	// Try to find root symbol info
	rootRes := t.findSymbolInfo(ctx, wctx.ID, symbolName)
	if rootRes != nil {
		rootNode.Type, _ = rootRes.Point.Payload["type"].(string)
		rootNode.FilePath, _ = rootRes.Point.Payload["file_path"].(string)
		rootNode.Package, _ = rootRes.Point.Payload["package"].(string)
	} else {
		// If nothing is indexed yet, ExactSearchPolyglot will return ErrNoCollectionsFound.
		// Signal indexing status instead of returning an empty hierarchy.
		_, sErr := t.engine.ExactSearchPolyglot(ctx, wctx.ID, map[string]interface{}{"name": symbolName}, 1)
		var noCollections *engine.ErrNoCollectionsFound
		if errors.As(sErr, &noCollections) {
			resp := ToolResponse{
				Status:  "indexing_required",
				Message: fmt.Sprintf("⏳ Workspace '%s' is not indexed yet. Indexing is required for complete call hierarchy results.", wctx.Root),
				Context: ContextFromWorkspaceWithStatus(wctx, t.engine),
			}
			if idx != nil {
				resp.Status = "indexing_in_progress"
				resp.Data = map[string]any{"indexing": idx}
			}