feat(dataset): add retrieval trace to dataset search response

[Ultronen]

Related issue

Closes #7113

Problem

When a RAG workflow returns a wrong or incomplete answer, the response detail only shows the surviving quotes. It does not show whether a chunk was never recalled, dropped during rerank, or removed by later filters.

Summary

Adds an optional retrievalTrace to dataset-search responses:

• Models text and image recall as separate parallel branches.
• Records the merged-candidate boundary and the post-merge deduplication, similarity, and token-limit pipeline.
• Marks optional rerank and Agent LLM chunk-selection stages as applied, skipped, or fallback.
• Reports score ranges only where scores are comparable; full-text and fused stages report counts only.
• Uses shared enums and Zod schemas, with static i18n keys in all four locales.
• Propagates the trace through both the standalone dataset-search node and the Agent RAG path.

Scope and compatibility

• Trace data contains statistics only and does not duplicate chunk content, so payload size stays bounded.
• The field is optional, preserving compatibility with existing chat records and providers that do not return a trace.
• Deep RAG is implemented by the separate Plus service. The open-source dispatcher now forwards an optional trace when that service returns one, but does not fabricate stages that it cannot observe.

Verification

• 28 focused tests passed across shared schema, default text/image recall, rerank success and fallback, standalone dispatch, Agent propagation and LLM selection, and UI SSR rendering.
• pnpm --filter @fastgpt/app typecheck
• Targeted ESLint and Prettier checks for all changed TypeScript files
• pnpm check:i18n-loaders
• git diff --check

[cla-assistant]

All committers have signed the CLA.

[cla-assistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

Ultronen seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[c121914yu]

Thanks for working on retrieval observability. I checked the diff at 33b18f9897 and reran the stated service test scope (6 files / 37 tests passed). Before merging, I think the following points need to be addressed or clarified:

1. The displayed stages are not a linear pipeline. textEmbeddingRecall, textFullTextRecall, and imageRecall are parallel branches; rerank only processes the text branch, and the text/image branches are merged afterward. For an embedding-only search with 2 recalled candidates and 1 surviving the similarity filter, the UI can show 2 → 0 → 0 → 2 → 2 → 1 → 1. Reading the counts top-to-bottom therefore does not identify where candidates disappeared. Please model/group recall branches separately from post-merge stages and include the merged-candidate boundary.

2. The implementation does not currently cover the main Agent RAG path. packages/service/core/workflow/dispatch/ai/agent/sub/dataset/index.ts discards retrievalTrace, then applies an additional selectRelevantChunksByLLM filter which is also untraced. Deep RAG also returns no trace. Please confirm whether this PR is intentionally limited to the standalone dataset-search node; if so, it does not fully close Feature: Retrieval Trace View for RAG Debugging #7113 as currently described.

3. Rerank state is ambiguous. A rerank stage is always emitted, including when rerank is disabled or the rerank request fails and falls back to the original candidates. The trace needs an applied/status indicator, or the stage should be omitted when it was not executed.

4. Full-text score ranges are not clearly comparable. Existing SearchScoreTypeMap sets fullText.showScore to false, and Mongo text scores / vector-store BM25 scores / scores from multiple queries are not normalized to one contract. Please omit the range for this stage unless its semantics are explicitly defined.

5. ResponseRows.tsx uses a dynamic translation key: t(\chat:response.retrieval_stage.${stage.name}` as any). FastGPT's i18n keys must remain statically discoverable; please replace this with literal-key branches/static calls. It would also help to define the stage names as a shared enum/schema instead of duplicating a free-form string` type in service and workflow schemas.

The current tests cover a simple embedding path and an empty result, but do not exercise mixed/image recall, rerank success/fallback, Agent propagation, dispatch persistence, or UI rendering. Those cases would catch the semantic issues above.

[Ultronen]

Thanks for the detailed review. I addressed the points in c1ac5241f:

1. Recall is now modeled as separate text and image branches, followed by an explicit merged-candidate boundary and a post-merge pipeline.
2. The Agent RAG path now propagates the default retrieval trace and appends its LLM chunk-selection stage, including success, skip, and failure-fallback behavior. Deep RAG is implemented by the separate Plus service, so the open-source dispatcher forwards an optional trace when returned but does not invent unobservable internal stages; the PR description now states this scope explicitly.
3. Rerank now reports applied, skipped, or fallback; failed rerank preserves the original candidates.
4. Full-text and fused stages no longer expose score ranges. Score bounds are emitted only for comparable embedding or applied rerank scores.
5. Stage and branch names now use shared enums and Zod schemas, and the UI maps every label through literal i18n keys.

I also added focused coverage for the shared/persisted schema, mixed text-image branches, rerank success and fallback, standalone dispatch, Agent propagation and LLM selection, and UI rendering. The 5 focused test files pass with 28 tests, together with app typecheck, targeted ESLint/Prettier, and the i18n loader check.

The branch is updated onto the latest main, and the commit identity now uses the GitHub-linked noreply address so the CLA check can associate it correctly.

[Ultronen]

Hi @c121914yu, following up now that c1ac524 addresses all five points from your review: branched recall plus an explicit merge boundary, Agent-path propagation and LLM selection tracing, explicit rerank status, comparable score ranges only, and shared typed/static-i18n stage names.

The branch is still mergeable, the CLA check passes, and the focused suite covers the shared schema, mixed text/image recall, rerank success/fallback, standalone and Agent dispatch, LLM selection, and UI rendering (28 tests). When you have time, could you take another look? I am ready to handle any follow-up quickly.