Repository navigation
fix(runtime): set explicit utf-8 encoding on subprocess text mode calls - #6140
huangruiteng merged 2 commits into
Conversation
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent; gpt-6.1-sol; OpenAI; runtime_reported; reasoning_effort=xhigh
[P2] 业务 stdout 不应静默替换损坏字节(loopx/capabilities/manager_context/ssh_evidence.py:235;同样影响 OpenVikingContextProvider._run)。真实子进程输出 JSON 中的 m、字节 0xff、n:基线 SSH 返回 remote_protocol_unavailable、coverage.complete=false,候选却返回 ok=true、title=m�n;OpenViking 从基线解码异常变为 completed、content=m�n。这不是修复本机编码选择,而是把损坏的事实改写成另一份成功结果。
最小修复:Keep encoding=utf-8 for structured stdout, reject malformed bytes and route decode failure through existing unavailable/contract-invalid paths. Decode only human diagnostic stderr/version text tolerantly where appropriate. Add real-byte positive and malformed-UTF8 negative tests; do not filter all literal U+FFFD because it can be valid source text.
Exact head: cd089e4aaf2844d3239206b4e84d64246649bfcc. Immutable comparison base: 40295867d72bf6a3403ab339d8a6f47073d00172.
动机
在非 UTF-8 默认环境中读取远端项目事实、检索已配置上下文,或检查本机 Lark CLI 可用性的用户。
同一子进程输出:候选能正确读回合法中文,但损坏字节也被替换成字符后返回成功;用户看不到读取失败,后续决策继续使用已经改写的事实。
真实子进程证明合法中文读取改善,同时证明 SSH 将损坏正文返回 ok=true,OpenViking 将其作为 completed item;本次可靠读取目标尚未达成。
本次只讨论三个既有子进程文本边界;不新增外部访问、授权、自动安装或语言迁移,也不把 Lark 版本诊断等同业务事实。
规范:spec_ref docs/reference/protocols/manager-evidence-and-continuity-v0.md;spec_revision 4029586;criterion_id “A prompt-only segment receives the declared source read”:失败读取必须显示 unavailable/stale 和 coverage gap,不能冒充当前进展;此 head 的损坏字节反例不满足该失败语义。
改动思路
保留三个现有 provider/transport owner 和显式 UTF-8;业务 stdout 严格解码,诊断输出按需要容错。两个读取 owner 的失败路由不同,不能用一条全局容错规则或新的通用框架代替。
这些 Python 模块承接既有外部 CLI/SSH I/O,归属合理;但业务事实和版本诊断具有不同的完整性约束,必须在各自入口保留该区分。
修复合法 UTF-8 在非 UTF-8 宿主上的解码,同时拒绝损坏的事实/上下文输出;Lark 版本探测可以保留有界诊断容错。
未来相关重构审查:继续使用现有 scanner/provider owner;本次没有需要新增的抽象、并行状态或语言迁移。最近的有效伴随修改是上述最小修复;不把后续猜测当作当前阻塞,也不根据六行或测试数量推断风险。已送达的恢复 RFC 经验不定义这里的验收;只采用普通任务、失败与结果读回的检查方法,不继承历史结论。
具体改动
三处都在既有 text=True 上增加 encoding 与 errors。显式 UTF-8 消除本机默认 code page 对合法协议字节的影响;replace 则在 JSON/schema 检查之前把无效字节变成 U+FFFD。JSON 仍合法,因此原本完整性失败不再进入 unavailable。OpenViking 的同一 _run 还用于读取、写入和不确定写入的核对,不能只按版本输出判断其风险。Lark 的 _version 只返回解析后的版本元数据,并保留有界超时和无本机路径的 public_snapshot,它与完整业务正文的约束不同。
关键代码讲解
-
loopx/capabilities/manager_context/ssh_evidence.py:152—read_remote:Configured host/grant before and after I/O; decode result -> JSON/schema -> scoped facts, now lossy accepted. -
loopx/capabilities/context_providers/openviking.py:294—OpenVikingContextProvider._run:Common real subprocess for preflight,search,read,sync; explicitUTF8 fixeslocale but replace changes bytes. -
loopx/capabilities/context_providers/openviking.py:382—OpenVikingContextProvider.retrieve:Validatednamespace/scope -> preflight -> boundedsearch/read -> exacttransient content; corruptbody now completed. -
loopx/extensions/lark/cli_resolution.py:106—_version:Five-second diagnostic probe,parsed version only; public snapshot excludes local executable path.
独立验证:Actual real subprocess and production parser return exact Chinese under US-ASCII at H; existing valid UTF8 branch preserved under UTF8locale. Four focused suites pass79tests at bothB/H, including grantrevocation/scope rejection.
负例与完整检查:Identical real child bytes at B/H, CPython3.14.8: valid Chinese under US-ASCII gives B SSH unavailable/OpenViking UnicodeDecodeError, H exact Chinese success. Under UTF-8 malformed byte0xff inside JSON title/content gives B SSH remote_protocol_unavailable with complete=false/no rows and provider UnicodeDecodeError; H SSH ok=true/title m�n and provider completed/content m�n. H does the same under US-ASCII. Real process/decoder/parser exercised; SSH endpoint and provider service synthetic.
运行来源为各自不可变基线/候选 worktree,CPython 3.14.8,验证导入来自该 checkout。Ruff 与 git diff --check 通过;开发期语义 advisory 先于完整检查运行,零候选只说明该探针范围。测试结果:四个聚焦文件,基线和候选各79项通过;两侧完整语义检查通过。没有查询、轮询或等待 GitHub CI。独立探针最初的调用脚本错误已纠正,失败记录保留;它们不是产品失败,也没有用于掩盖上面当前候选的失败。
对主干的风险
Malformed provider/remote bytes remain syntactically valid JSON after replacement and are consumed as current facts/content without a decode failure. 用户的普通路径是继续调用原有命令或已授权读取,没有新增确认或重复配置步骤。No live SSH host, paid provider or Lark account was contacted; only real local children and actual LoopX parsing owners. Native Windows code page not executed; actual US-ASCII default locale demonstrates dependency removal. Sync/reconciliation callers share _run; lossy business-content acceptance blocks this exact head. 上述测试没有扩大权限,合成注册表、临时配置与子进程均不修改活动 Goal 或真实外部服务。正例不能抵消候选的关键失败。既有 schema、quota、scheduler 与授权边界未增加;评审结论也不授予合并权限。恢复必须由原有内容/读取 owner 完成,纠正后重新读回,不能把收到成功字段当作完整性证明。
我的整体评价
REQUEST_CHANGES。ExplicitUTF8 fixes locale handling, but unconditional replacement widens protocol acceptance and falsifies successful readback; concrete current regression. Keep encoding=utf-8 for structured stdout, reject malformed bytes and route decode failure through existing unavailable/contract-invalid paths. Decode only human diagnostic stderr/version text tolerantly where appropriate. Add real-byte positive and malformed-UTF8 negative tests; do not filter all literal U+FFFD because it can be valid source text. 修改范围本身合理,原有 owner 可承接;当前阻塞都有基线/候选差异和可重跑证据。修复后应在新精确 head 重做相关失败检查,并保留本次已通过的合法路径与反方向负例,不能仅改评审标签或削弱检查。
English verdict: REQUEST_CHANGES. Explicit UTF-8 correctly fixes valid Chinese output under a non-UTF-8 parent locale, but errors=replace silently changes malformed structured stdout into accepted evidence/content. A real child writing invalid byte0xff inside a JSON title makes the base report remote_protocol_unavailable with incomplete coverage; this head returns ok=true with m�n. OpenViking similarly returns completed corrupted content. Keep strict UTF-8 for business stdout, handle decode failures through the existing failure owners, and reserve tolerant decoding for bounded human diagnostics. Add actual-byte positive and malformed-input regression tests; valid literal replacement characters must not be blanket-rejected.
Validation: Actual real subprocess and production parser return exact Chinese under US-ASCII at H; existing valid UTF8 branch preserved under UTF8locale. Four focused suites pass79tests at bothB/H, including grantrevocation/scope rejection. Identical real child bytes at B/H, CPython3.14.8: valid Chinese under US-ASCII gives B SSH unavailable/OpenViking UnicodeDecodeError, H exact Chinese success. Under UTF-8 malformed byte0xff inside JSON title/content gives B SSH remote_protocol_unavailable with complete=false/no rows and provider UnicodeDecodeError; H SSH ok=true/title m�n and provider completed/content m�n. H does the same under US-ASCII. Real process/decoder/parser exercised; SSH endpoint and provider service synthetic. No live SSH host, paid provider or Lark account was contacted; only real local children and actual LoopX parsing owners. Native Windows code page not executed; actual US-ASCII default locale demonstrates dependency removal. Sync/reconciliation callers share _run; lossy business-content acceptance blocks this exact head.
Set encoding='utf-8' and errors='replace' on subprocess text-mode invocations in openviking, ssh_evidence, and lark cli_resolution to prevent locale code page misdecoding or UnicodeDecodeError on Windows. Signed-off-by: mika <spacexstarship01@outlook.com>
- Remove silent byte replacement on structured stdout in SSH evidence and OpenViking - Preserve tolerant replace decoding for diagnostic CLI version probe - Route decode failures through existing unavailable/protocol failure paths - Add test coverage for valid UTF-8 Chinese stdout and malformed UTF-8 rejection Signed-off-by: mikamikasuki <211269698+mikamikasuki@users.noreply.github.com>
cd089e4 to
7a1c1eb
Compare
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent · gpt-6.1-sol · OpenAI · runtime_reported · reasoning_effort=max
APPROVE。旧评审指出的“损坏业务字节被改写成成功事实”已在当前精确 head 修复,合法 UTF-8 的原始目标也通过真实消费者验证。未发现阻塞问题。
动机
在非 UTF-8 系统中读取远程项目事实、检索已配置上下文,或探测 Lark CLI 版本的用户。
同一个子进程已经输出正确的 UTF-8 中文,旧入口仍按本机字符集解码,导致读取失败或版本探测抛异常;用户得不到原本可用的内容。
当前提交能按原文读回合法中文,同时拒绝损坏的业务输出;只有人类诊断用的版本文本允许替换损坏字节。
本次仅修复三个既有文本边界,不新增远程访问、账号权限、自动安装、状态协议或用户配置步骤。
改动思路
业务 stdout 的原文完整性和人类版本诊断的容错具有不同要求;在原有 SSH、OpenViking 与 Lark owner 内分别处理,比引入全局宽松解码器或新框架更小且更准确。
当前 PR 交付显式 UTF-8 的有效内容读取、损坏业务输出的既有失败路由,以及版本诊断的有界容错;不扩展成 provider、持久状态或整个平台迁移。
先让真实子进程产生字节,再由生产读取函数和后续消费者作判断;不能用已解码字符串或直接抛异常的替身证明编码有效。当前 SSH 仍检查授权和返回结构,OpenViking 的读取错误由现有上下文消费者转成 unavailable,Lark 只提取版本元数据。关闭上下文 provider 时仍返回原来的无检索结果,没有自动启用外部服务或增加确认轮次。
| 输入 | 当前业务读取 | 当前版本诊断 |
|---|---|---|
| 非 UTF-8 宿主上的合法中文 | 原文成功读回 | 正常取得版本 |
| JSON 内容含非法字节 | unavailable,不返回损坏正文 | 诊断可容错后提取版本 |
| 合法文字本来包含 U+FFFD | 保留该合法字符 | 正常解析 |
表格对应本次真实字节探针,不代表原生 Windows 或在线服务验收。
具体改动
关键代码讲解
完整复审 head 7a1c1eb105e7da2a2490ca5de06d936af9ccb637,实际合并基线 5706ef0552d5c22af805f264ae891556a0f819cf。全部差异为五文件 +77/-0:三个生产调用增加四行参数,另两文件加入三个回归测试。相对旧评审 cd089e4 的修复与当前整头分开检查;基线、调用方和当前测试都重新核实,没有继承旧结论。
- read_remote 在原子进程调用明确 UTF-8,保持严格解码;UnicodeDecodeError 属于既有 ValueError 路由,因此输出损坏时返回协议不可用,授权改变后也不会泄露读到的行。
- OpenVikingContextProvider._run 为已有版本、搜索、读取与同步传输指定同一编码,移除了旧头的 errors=replace。collect_context_recall 的真实调用证实:损坏正文不会成为 completed 内容,而会进入现有失败分支。
- _version 保留五秒上限和纯版本读回,仅诊断文本采用替换容错;可执行文件发现与账户授权规则没有改变。
规范依据:spec_ref docs/reference/protocols/manager-evidence-and-continuity-v0.md,spec_revision 5706ef0552d5c22af805f264ae891556a0f819cf,criterion_id A prompt-only segment receives the declared source read。该原有条款要求失败来源保持 coverage gap,不冒充当前进展,也不拖垮整个 Turn。本头通过既有 SSH 不可用结果及真实上下文消费者的 unavailable 读回满足它;普通编码修复和诊断范围还沿原 PR 请求及旧评审判定,没有新增全仓义务。
对主干的风险
我在 CPython 3.12.15 的 US-ASCII 和 UTF-8 父进程环境中,使用相同真实子进程字节检查 ASCII、中文、合法 U+FFFD、非法 0xff、撤销、越界来源与关闭路径:当前两种环境各15项均满足预期,基线分别8/15与14/15。另补同一主机的未授权 Goal、新增未授权结果及拒绝后恢复的实际读取,两侧都保持范围边界。没有连接真实 SSH、云端 OpenViking 或 Lark 账户,也未执行原生 Windows code page。
四个聚焦文件当前90项、基线87项通过;source canary premerge 的6项直接检查和16项选中检查通过,DCO与差异检查通过,没有查询或等待CI。新加的部分作者测试用替身输出字符串或直接抛异常,不能单独证明真实解码;本次独立字节验证补足了这点。可将其中一个现有测试换成真实字节回归,属于非阻塞的覆盖改善,不需要新抽象或新测试框架。没有新增状态词表、prose 分类规则或自动加载指令。
我的整体评价
APPROVE。long_horizon 与 user_experience 在这条既有读取链上改善:合法内容可继续使用,损坏内容保持明确失败,恢复后原入口继续返回合法结果;没有增加人工准备或确认。改动复用原 owner,生产成本与缺陷相称,也没有以编码修复引入另一套兼容协议。该判断只覆盖当前源码和本次真实本地传输验证,不宣称安装态、多平台或长期净收益,也不授予 dismissal 或 merge 权限。
English verdict: APPROVE — head 7a1c1eb. Explicit UTF-8 fixes real non-UTF-8-parent failures while structured SSH/context output remains strict; malformed bytes become unavailable and valid literal replacement characters survive. Only bounded version diagnostics are tolerant. Real local-child consumer and scope/recovery controls,90 focused head tests and the native risk gate passed; native Windows and live services remain unqualified.
Set encoding='utf-8' and errors='replace' on subprocess text-mode invocations in openviking, ssh_evidence, and lark cli_resolution to prevent locale code page misdecoding or UnicodeDecodeError on Windows.
Goal And Delivered Outcome
Outcome basis / optional anchor:
Goal/source and gap:
Observable before → after, with the validation row that proves it:
Issue/task and intended base:
Author Declaration
Implemented against
Scope And Continuation
Validation
unitnot_runSee validation disclosure guidance.
Frontend / Visual Evidence
Type of Change
LoopX Area
Technical Direction
Shared-authority RFC fixture impact
Boundary Checklist
.loopx/,.codex/goals/, and liveACTIVE_GOAL_STATE.md).none.Signed-off-bytrailer (git commit -s).