fix(runtime): surface OpenAI and Claude reasoning summaries - #2645
fix(runtime): surface OpenAI and Claude reasoning summaries#2645hqhq1025 wants to merge 1 commit into
Conversation
hqhq1025
changed the title
0b82b9c to
c87bef5
Compare
|
Self-review completed after marking ready. Fixed two issues before merge:
Rebased onto the latest |
|
Thanks — the gap is real (on main, Conclusion: PASS — all P3, including one worth handling. P3-1 (worth handling): explicit P3-2: the Anthropic response path for AI-assisted review disclosure: this review was produced with AI assistance (pi review subagent on 中文摘要（AI 辅助审查）结论：PASS，全 P3（其中 P3-1 建议处理）。问题真实（main 上 buildProviderOptions 在用户未选 thinking 级别时对 openai/anthropic/claude-subscription 一律不发 reasoning 参数 → 模型按默认运行、reasoning 不出现在 transcript；display 链路本就存在——SDK reasoning_summary_text.delta → reasoning-delta → adapter thinking 事件 → 既有 thinking_delta/AssistantMessage.thinking），改动补齐请求端默认，非并行实现。已按固定版本 SDK 源码验证 wire 行为：reasoningSummary:'auto' 发 reasoning:{effort,summary:'auto'} 且 store:false 时仍带 reasoning.encrypted_content（与 #2328/#2518 加密链共存）；thinking:{type:'adaptive',display:'summarized'} 由 SDK 原样透传。无 schema/协议改动，旧数据/旧客户端不受影响，未被 main 后续提交取代。P3-1（建议处理）：显式 off 在 Claude effort 模型上被静默升级为开启可见 adaptive thinking，与 PR 自身政策矛盾（PR 自带测试断言 off→{thinking:{type:'adaptive',display:'summarized'}}）——旧行为是发空配置；仓库惯例（kimi-coding-plan 分支"必须大声失败、不得静默升级"）是拒绝而非升级。当前仅直接 runtime 调用方可达（UI 对 effort 模型不提供 off 选项），建议显式拒绝或如实文档化。P3-2：Anthropic 响应路径无 dedicated 测试（只有请求体断言）；P3-3：默认开启的 wire 参数对第三方 relay 无能力协商（代理旧版 API 可能 400 或静默丢字段），合并前建议对真实 relay 实测；P3-4：分支落后 main ~530 commits（CONFLICTING），"3325 passed" 是对旧树的验证，rebase 后重跑全套。 |
Astro-Han
left a comment
Astro-Han
left a comment
There was a problem hiding this comment.
The main design is sound: this completes the existing provider-options → SDK → thinking-display chain rather than creating another transcript path, and the provider/wire split has broad focused coverage.
This head is now conflicting with current main in two reasoning-contract test files, so its old-base green checks are not integration evidence; please rebase and rerun the focused Runtime tests, typecheck, and CI. One additional low-severity edge remains for custom Chat relays: an explicit unsupported setting can be silently replaced by the GPT-family default. From first principles, an invalid explicit choice must not become a different valid choice; defaulting should apply only when no choice was supplied.
Reviewed with Codex using two independent reviewer agents and an external DeepSeek review; I verified the latest head, current-main conflicts, provider-option paths, existing discussion, and CI state.
中文
主方案是合理的：它补齐现有 provider-options → SDK → thinking-display 链路，没有建立第二条 transcript 路径；provider/wire 分流也有较完整的聚焦测试。
但当前 head 已与最新 main 在两个 reasoning contract 测试文件中冲突，旧 base 上的全绿检查不能证明可集成；请先 rebase，再重跑 Runtime 聚焦测试、typecheck 与 CI。另有一个低严重度边界：自定义 Chat relay 的显式不支持选项可能被静默替换成 GPT family 默认值。按第一性原理，无效的显式选择不应变成另一个有效选择；只有调用方没有提供选择时才应使用默认值。
本次由 Codex 配合两个独立 reviewer agent 和外部 DeepSeek 审查；我核验了最新 head、current-main 冲突、provider option 路径、已有讨论和 CI 状态。
| } | ||
| if (wire === 'openai-chat' && adapter.kind === 'openai-compatible') { | ||
| const reasoningEffort = explicitReasoningEffort ?? defaultOpenAiReasoningEffort(modelId); | ||
| if (reasoningEffort) { |
There was a problem hiding this comment.
P3 — Do not replace an invalid explicit relay choice with medium. For an undeclared custom OpenAI Chat relay, resolveThinkingLevel can discard an explicit off/high, after which this fallback recognizes a GPT-family model id and emits reasoningEffort: 'medium'. Direct Runtime callers can therefore ask to disable thinking and instead enable it, or ask for high and get downgraded. Apply the family default only when the original thinkingLevel was absent; if an explicit level resolves unsupported, omit options or reject it. Add custom-relay off/high coverage.
|
/agentic_review |
Code Review by Qodo
1. Copilot omits default effort
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
| const reasoningEffort = | ||
| explicitReasoningEffort ?? | ||
| (adapter.kind === 'openai' ? defaultOpenAiReasoningEffort(modelId) : undefined); |
There was a problem hiding this comment.
1. Copilot omits default effort 🐞 Bug ≡ Correctness
Fix-now: a GitHub Copilot model declared with apiProtocol: 'openai-responses' retains adapter kind github-copilot, so an unset thinking level produces summary: 'auto' without the default `effort: 'medium'`, violating the resolved Responses-wire request contract. The adapter-kind guard incorrectly restricts default effort selection to native OpenAI adapters even though Copilot resolves to the same wire.
Agent Prompt
## Issue description
`buildFamilyWire` restricts default GPT reasoning-effort derivation to adapters whose kind is `openai`. GitHub Copilot models can resolve to the OpenAI Responses wire while retaining adapter kind `github-copilot`, causing an unset thinking level to emit `summary: 'auto'` without the required default `effort: 'medium'`.
## Issue Context
Treat the resolved wire as the routing authority and keep using the native OpenAI provider-options namespace for the Responses wire. Reuse `defaultOpenAiReasoningEffort(modelId)`, which already returns a value only for recognized OpenAI reasoning model IDs, rather than adding a Copilot-specific branch, new state, configuration, public API, or test seam. Extend coverage beyond the existing explicit `high`-level Copilot case to verify the unset-level default.
## Fix Focus Areas
- packages/runtime/src/model-factory.ts[593-605]
- packages/runtime/src/__tests__/model-factory-thinking.test.ts[327-340]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| const familyModelId = modelId.includes('/') | ||
| ? modelId.slice(modelId.lastIndexOf('/') + 1) | ||
| : modelId; | ||
| if (!familyModelId.startsWith('claude-')) return undefined; | ||
| const effectiveOptions = thinkingOptions ?? thinkingOptionsForModel('anthropic', modelId); |
There was a problem hiding this comment.
2. Namespaced claude loses thinking 🐞 Bug ≡ Correctness
Fix-now: visibleClaudeThinking recognizes the final segment of a namespaced Claude ID but looks up fallback metadata using the unnormalized ID, so an Anthropic-compatible route such as anthropic/claude-opus-4-8 emits no thinking options. This prevents the default summarized thinking promised for recognized Claude models.
Agent Prompt
## Issue description
Namespaced Claude model IDs pass the family check but miss Anthropic metadata, disabling the new default visible-thinking request.
## Issue Context
`familyModelId` is already the normalized family identifier. Reuse it for the fallback metadata lookup; no new state, branch, configuration, or public surface is needed.
## Fix Focus Areas
- packages/runtime/src/model-factory.ts[367-385]
- packages/runtime/src/__tests__/model-factory-thinking.test.ts[343-354]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
jackwener
left a comment
jackwener
left a comment
There was a problem hiding this comment.
Automated review of exact head c87bef5a51f8d74e70d763e8d96f7f7cfd93fd5f against current main@d62857a8357e9160926726a2a13096bc2dc2b91d.
The bug still exists on current main: recognized GPT Responses routes do not request reasoningSummary, and recognized Claude Anthropic routes do not request summarized display by default. The PR's requested default-medium/summarized policy matches issue #2644, and its request-shape plus real SSE boundary coverage is directionally strong.
Blocking rebase/authority conflict. This head is based before the current Responses routing and replay contract. Current buildFamilyWire distinguishes native OpenAI from open-responses, verifies reasoningReplay, keys provider options by the actual Open Responses provider name, preserves serviceTier, and enables forceReasoning for encrypted-content replay. The PR's version destructures only { adapter, wire } and returns a single openai namespace for every Responses wire. It is now text-conflicting, and resolving that mechanically would either drop current replay/service-tier behavior or silently send summary options to a namespace the Open Responses adapter does not read.
Rebase and implement visibility at the current resolved-wire seam: retain the existing reasoningReplay and open-responses branches, add OpenAI reasoningSummary: 'auto' only where the native OpenAI Responses schema consumes it, and preserve provider-native options/names for Open Responses relays. Re-run the real Responses SSE test and the provider contract sweep on the rebased exact head.
Required conclusions:
- Optimal for the actual problem: the policy is sound, but this revision is no longer an optimal/current implementation.
- Production code to delete: none identified beyond avoiding a parallel/legacy Responses branch during rebase.
- Tests to delete/replace: none identified; port the strong request-shape and SSE tests to the current routing matrix.
- Deeper refactor: no new abstraction; adapt the change to the existing current-main wire/replay authority.
- Ready to merge: no; it is conflicting and predates material routing contracts.
- Residual risks/gaps: provider-visible reasoning changes user-visible behavior and token/cost characteristics, so independent human review and exact-head CI remain required after rebase.
c87bef5 to
ebfab4b
Compare
|
Rebased onto current This revision addresses the review findings while preserving the current runtime routing contracts:
Validation on the exact head:
Ready for re-review. |
3 participants
Closes #2644
Default policy
Reasoning-capable GPT and Claude models request user-visible thinking by default. Thinking is disabled only when the user explicitly selects
off. Routing follows the resolved wire and recognized model family, not the provider label.OpenAI / GPT
reasoning: { effort: "medium", summary: "auto" }for recognized GPT reasoning modelsreasoning_effort: "medium"and keep using the existingreasoning_contentresponse transporteffort: "none"and omits summaryClaude / Anthropic
thinking: { type: "adaptive", display: "summarized" }thinking: { type: "disabled" }where supportedThinking display integration
No new Desktop component is required. Returned provider reasoning follows the existing chain:
reasoning-deltaModelAdapterMakathinkingeventthinking_delta/thinking_completeAssistantMessage.thinkingThe PR includes a real OpenAI Responses SSE contract test proving summary deltas become non-empty Maka thinking text. Existing signed-thinking tests cover the Anthropic delta/signature path, while request-shape tests cover direct and third-party GPT/Claude routes.
Validation