feat(retrieval): 支持 Agentic Search 三维度评测及内置中文检索数据集 - #495
Open
pekopoke wants to merge 2 commits into
Open
Conversation
背景 在现有 Meta Search 搜索结果评测基础上,扩展 Agentic Search 主观评测,并内置中文检索数据集 cjk_evalset_v1,补充中文检索场景的评测能力。 主要改动 1. Agentic Search 主观评测 通过 Dingo LocalExecutor 执行三个维度的评测: - 相关性:基于 query、title 和 chunk 进行 LLM 判断。 - 有效性:title、abstract、chunk、source 四项各占 0.25;结合规则初筛与异常候选 LLM 二次确认。 - 权威性:通过 doc_id 查询 Meta Search 元数据,补充引用、期刊、出版社及 DOI 等评分依据。 原文核验使用返回的 doc_id + offset 请求 /content,默认比较前 50 字符,仅归一化换行符,不进行邻近 offset 校准,并保留核验文本及状态。有效性必要证据缺失或复核失败时保留空分。 2. 内置中文检索数据集 新增 cjk_evalset_v1,包含 100 条中文 query、分组及源文档 doc_id 标注
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
在现有 Meta Search 搜索结果评测基础上,扩展 Agentic Search 主观评测,并内置中文检索数据集 cjk_evalset_v1,补充中文检索场景的评测能力。 主要改动