feat(dataset): add synonym management and search enhancement

[lean-zone]

What does this PR do?

This PR adds dataset-level synonym management and integrates synonym normalization into data indexing, retrieval, reranking, and citation metadata.

Each dataset can maintain one active synonym configuration. Synonym mappings are stored as versioned MongoDB snapshots and can be managed through JSON mappings or uploaded files.

Main changes

• Add dataset synonym upload, update, download, delete, detail, and paginated mapping APIs.
• Support CSV, XLSX, and XLS files:
  • The first column contains the standardized term.
  • Remaining columns contain its synonyms.
• Add a synonym management page with configuration details, mapping search, pagination, and rebuild progress.
• Add OpenAPI schemas and Simplified Chinese, Traditional Chinese, and English translations.
• Preserve the original chunk content and indexes.text values.
• Apply synonym replacement only to derived vector and full-text index inputs.
• Normalize synonyms during:
  • Data creation
  • Data editing
  • System index rebuilding
  • Synonym configuration rebuilds
• Expand retrieval queries without increasing the number of recall calls.
• Include synonym mapping context in reranking inputs.
• Include matched chunk-level synonymMappings in existing knowledge-base citation metadata.
• Preserve existing behavior for datasets without an active synonym configuration.

Rebuild and consistency behavior

• Store immutable, versioned mapping snapshots in MongoDB.
• Use fileVersion to associate mappings with a complete configuration snapshot.
• Only the current configuration snapshot is used at runtime; historical mapping versions are removed after a successful update.
• A synonym update or deletion triggers a full text-scoped rebuild of existing dataset data.
• Synonym rebuilds reuse the existing dataset rebuild queue and training structure.
• Text-scoped rebuilds regenerate text indexes and full-text derived data while preserving existing image indexes and image embeddings.
• The active synonym configuration becomes available immediately after the configuration transaction commits.
• Rebuild processing is asynchronous, so mixed old/new vectors may exist temporarily. The system converges to the current synonym configuration after rebuild completion.
• Mapping writes are split into bounded batches.
• Existing dataset data is marked for rebuild in bounded batches outside the configuration transaction to avoid oversized transactions on large datasets.
• If seed task creation fails after configuration commit, the affected data remains marked as rebuilding so the existing queue recovery and retry flow can continue.
• If a rebuild queue encounters data whose collection no longer exists, it skips that item without deleting the underlying dataset data.
• Concurrent synonym updates use configuration version checks and reject stale update requests.
• Data writes retain synonym snapshot checks before committing derived indexes, but do not use dataset.updateTime as a dataset-wide write lock.

Permissions and billing

• Read operations require dataset read permission.
• Upload, update, and delete operations require dataset write permission.
• Embedding work caused by synonym changes uses the existing dataset training billing pipeline.
• Synonym rebuilds use the existing text rebuild scope and training task model.

Migration

A Mongo-only migration script is included:

pnpm --filter @fastgpt/app migrate:dataset-synonym-mongo-only

The script runs in dry-run mode by default. Pass --execute to apply changes.

The migration converts legacy synonym configuration and mapping data into the current schemaVersion=2 snapshot format and is designed to be safely rerunnable.

The migration should be run during downtime with application and worker processes stopped. Legacy S3 files are not deleted.

The new synonym APIs no longer use legacy S3 fileId or pendingFileId fields. Existing synonym data must be migrated before enabling the new implementation.

Limits and matching behavior

• Maximum file size: 10 MiB
• Maximum mappings: 10,000
• Maximum terms: 50,000
• Maximum term length: 128 Unicode code points
• Maximum total term length: 500,000 Unicode code points
• Longest matching term takes precedence.
• English and numeric terms use case-insensitive word-boundary matching.
• Chinese terms support continuous substring matching.
• Replacement results are not matched recursively.
• Conflicting mappings, cycles, and cross-group term reuse are rejected during normalization.

Verification

• App TypeScript typecheck passed.
• ESLint and Prettier checks passed.
• Targeted Vitest suites cover:
  • Synonym parsing and normalization
  • Dataset search and reranking integration
  • Data creation and index rebuilding
  • Rebuild queue behavior
  • Synonym mutation and migration behavior

The local targeted Vitest run may require access to download the MongoDB binary used by mongodb-memory-server.

Design document

The detailed data model, matching rules, rebuild flow, migration strategy, and compatibility constraints are documented in:

.agents/design/core/dataset/synonym-feature-design.md

[cla-assistant]

All committers have signed the CLA.

[cla-assistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

zhanglin seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[c121914yu]

混杂了知识库队列优化和同义词两个需求，需要单独拆分。并且现在改的队列方式也不是完全可靠的。
如果只是单独做同义词的话，简单复用现在队列就行，可以接受重构期间搜索有问题。

[c121914yu]

不复用 training 表来预处理么

[github-actions]

✅ Build and publish successful - Preview fastgpt Image:

ghcr.io/labring/fastgpt-pr:fastgpt_d7c5d15095b9e68cd65e2456d626be1af40faf31

🕒 Time: 2026-09-05 11:18:41 (UTC+8)

[c121914yu]

这期建议先收敛范围，只实现同义词能力，不在这个 PR 里同时建设一套新的训练/重构队列。

具体调整建议：

1. 同义词规则更新后直接生效，允许重构期间知识库中存在新旧 embedding 混合、召回短暂不一致。
2. 复用现有 dataset_trainings 作为重构任务队列：遍历 data 的 q/a 和所有非图片 index，命中本次变化词的数据写入 training，由现有 worker 原地更新 embedding。
3. 本期不引入同义词专用的幂等 saga、自动回退、active/pending 双版本和独立 mutation lock；失败任务继续使用 trainings 现有的重试/人工重触发能力。
4. 重构期间先禁止再次修改同义词，避免两次规则更新互相覆盖。这个状态可以直接放在同义词配置记录里，不需要额外的 job/lock 集合。
5. 原始 q/a/index text 保持不变，同义词转换只用于 embedding 输入，确保后续仍可按最新规则重新生成。

因此数据结构可以先简化为同义词配置/映射集合，加已有的 dataset_trainings。当前的 dataset_synonym_jobs、dataset_synonym_operations、dataset_mutation_locks，以及版本切换和自动 rollback 逻辑建议从本期移除。

训练/重构队列的通用可靠性问题——例如任务幂等、租约续期、崩溃恢复、向量写入补偿、计费去重和统一进度模型——后续单独开需求统一优化，避免只为同义词做一套专用实现。

[c121914yu]

补充本次 review 发现的其他问题。即使按上一条建议简化方案，下面第一组仍需要处理。

简化后仍需修复

1. 多知识库搜索可能丢失原始 query，导致未配置同义词的知识库召回下降

   packages/service/core/dataset/search/utils.ts:112 目前只在某个知识库存在 pending 时设置 preserveOriginal。例如同时搜索 A、B，只有 A 配置“退钱 -> 退款”时，最终会把“退钱”直接替换成“退款”，B 中只包含“退钱”的数据就无法按原词召回。

   多知识库联合搜索时，需要确认所有目标知识库都适合替换后才能删除原词；更简单的做法是只要搜索多个知识库就始终保留原词。按本期允许新旧 embedding 混合的方案，重构期间也必须保留原词。

2. 直接 JSON API 接受纯空白词条

   packages/global/core/dataset/synonym.ts:210-212 的 schema 只使用 z.string()。文件上传路径会先 trim/filter，但 JSON mappings 可以传入 standardizedTerm: " " 或空白 synonym；之后 normalize 会得到空字符串，可能形成“词条映射为空输出”。

   建议在 API schema 和 normalize 后都保证 trim().min(1)，synonym 数组中的每一项也需要校验。

3. 标准词仅修改大小写时被误判为 unchanged

   packages/service/core/dataset/synonym/utils.ts:296-298 的 fingerprint 只包含 normalizedStandardizedTerm。例如标准输出从 OpenAI 改为 openai，实际 embedding/fulltext 输入已经变化，但 fingerprint 不变，不会标记相关数据重构。

   fingerprint 需要包含最终实际输出的 standardizedTerm；或者明确规定标准词大小写不影响输出并统一输出规范。

4. 映射管理页最多只显示前 100 条

   projects/app/src/pageComponents/dataset/detail/Synonym/index.tsx:88-94 固定请求 pageNum: 1, pageSize: 100，但后端允许最多 10000 组，页面也没有翻页或继续加载，剩余映射不可见。需要接入分页组件或滚动加载，并使用接口返回的 total。

5. 动态 i18n key 无法被静态扫描

   projects/app/src/pageComponents/dataset/detail/Synonym/index.tsx:295 使用 t(\dataset:synonym.status_${currentJob.status}` as any)`，绕过了类型和静态 key 扫描。建议建立显式的 status -> i18n key 映射。

如果仍保留当前复杂队列方案，还存在以下阻塞问题

1. 独占租约过期后 processing job 无法恢复

   projects/app/src/service/core/dataset/queues/synonym.ts:182-188 只会 renew，renew 又要求旧租约尚未过期。进程暂停超过 5 分钟后，所有 training 会持续失败；现有启动恢复只恢复 marking，不会为 processing/rollingBack job 重新 acquire 锁，最终 rollback 也会因旧锁失效而无法完成。

2. rollback 与在途正向 worker 存在提交竞态

   packages/service/core/dataset/synonym/controller.ts:756-770 切换到 rollingBack 时没有撤销/递增 fencing token；而正向 worker 的 beforeCommit 只检查锁和 token，不检查 job 仍为 processing。已经领取任务的 worker 可以在 rollback 扫描之后继续提交 pendingVersion，导致回退遗漏数据。

3. operation 信息落库失败会遗失新向量

   projects/app/src/service/core/dataset/data/data.ts:337-343 的 onVectorsPrepared 位于向量写入之后、清理 try/catch 之外。如果 operation update 失败，刚写入的新 vector id 和 token 既不会被回收，也没有记录可供 reconciler 对账。

4. collection sync 绕过 mutation gate

   projects/app/src/pages/api/core/dataset/collection/sync.ts:32 直接调用 syncCollection，而同步过程会替换 collection 并删除旧 data/training/vector。当前其他修改/删除入口增加了 mutation gate，但这里没有，可能与同义词扫描或 worker 并发破坏锁假设。

如果按上一条评论删除专用 operation、rollback 和 mutation lock，上述第二组不需要逐项修补；应随相关实现一起移除。

[ctlaltlaltc]

导入和 rebuild 尽量简单、复用原结构。

但当前mutation 自己实现了配置锁、版本快照、全量扫描、批量入队、失败清理和 matcher 缓存。尤其训练队列方面，当前没有充分处理与原有训练任务并存的情况。mutation 复用了旧训练队列 worker，但没有完全复用旧的训练任务创建/rebuild 编排结构，保持精简