Skip to content

fix(runtime): set explicit utf-8 encoding on subprocess text mode calls - #6140

Merged
huangruiteng merged 2 commits into
loopx-project:mainfrom
mikamikasuki:codex/subprocess-utf8-encoding
Oct 21, 2026
Merged

huangruiteng merged 2 commits into
loopx-project:mainfrom
mikamikasuki:codex/subprocess-utf8-encoding

Conversation

@mikamikasuki

Copy link
Copy Markdown
Contributor

Set encoding='utf-8' and errors='replace' on subprocess text-mode invocations in openviking, ssh_evidence, and lark cli_resolution to prevent locale code page misdecoding or UnicodeDecodeError on Windows.

Goal And Delivered Outcome

  • Outcome basis / optional anchor:

  • Goal/source and gap:

  • Observable before → after, with the validation row that proves it:

  • Issue/task and intended base:

Author Declaration

  • Written by:

Implemented against

  • Specification and revision:
  • Criteria:
Criterion (spec clause) Disposition Symbol / path Test or command
  • Self-check before submission:

Scope And Continuation

  • Completed scope and remaining work:
  • Slice boundary / successor:

Validation

Public-safe summaries only, including in HTML comments. Do not paste raw logs,
private data or prompts, private screenshots, credentials, internal URLs, connection
strings, or local paths. Use repository-relative commands without sensitive
arguments, aggregate results, and already-public CI links. If evidence is
private, report its category and limitations only; do not upload it to prove a claim.

  • Tested revision:
  • Run state:
  • Input classes:
Check kind Result Public-safe evidence / limitation
unit not_run Example only: repository-relative test path; behavior checked; aggregate outcome or generic reason not run.
  • Coverage and gaps:

See validation disclosure guidance.

Frontend / Visual Evidence

  • UI impact:
  • Before:
  • After:
  • States and viewports shown:
  • Source data:
  • Attention review:

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Refactoring (no functional changes)
  • Documentation update
  • Test update

LoopX Area

  • Control plane (goals, todos, quota, scheduler, registry, runtime)
  • Benchmark boundary (adapters, runners, verifiers, scoring, evidence)
  • Capability or extension (providers, adapters, skills)
  • Public docs or presentation surface (README, protocols, dashboard)
  • Build, packaging, installer, or CI
  • Host or runtime integration

Technical Direction

  • Direction / acceptance reference, when applicable:

Shared-authority RFC fixture impact

  • Production-scale fixture schema:
  • Semantic dimensions changed, or reviewed no-impact rationale:
  • Provider conformance arms run:
  • Read-only legacy/file/PostgreSQL three-arm rehearsal (required for promotion, runtime-routing, or compatibility-projection changes):

Boundary Checklist

  • Neither the diff nor this PR body/comments/attachments disclose private state, credentials, raw traces or verifier output, internal links, or local machine paths (including .loopx/, .codex/goals/, and live ACTIVE_GOAL_STATE.md).
  • I did not duplicate maintainer-owned benchmark work unless a maintainer split out a public issue for it.
  • I kept the change scoped to the linked issue/task.
  • I completed the visual evidence section for UI changes, or marked UI impact none.
  • Every commit includes a DCO Signed-off-by trailer (git commit -s).

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent; gpt-6.1-sol; OpenAI; runtime_reported; reasoning_effort=xhigh

[P2] 业务 stdout 不应静默替换损坏字节(loopx/capabilities/manager_context/ssh_evidence.py:235;同样影响 OpenVikingContextProvider._run)。真实子进程输出 JSON 中的 m、字节 0xff、n:基线 SSH 返回 remote_protocol_unavailable、coverage.complete=false,候选却返回 ok=true、title=m�n;OpenViking 从基线解码异常变为 completed、content=m�n。这不是修复本机编码选择,而是把损坏的事实改写成另一份成功结果。

最小修复:Keep encoding=utf-8 for structured stdout, reject malformed bytes and route decode failure through existing unavailable/contract-invalid paths. Decode only human diagnostic stderr/version text tolerantly where appropriate. Add real-byte positive and malformed-UTF8 negative tests; do not filter all literal U+FFFD because it can be valid source text.

Exact head: cd089e4aaf2844d3239206b4e84d64246649bfcc. Immutable comparison base: 40295867d72bf6a3403ab339d8a6f47073d00172.

动机

在非 UTF-8 默认环境中读取远端项目事实、检索已配置上下文,或检查本机 Lark CLI 可用性的用户。

同一子进程输出:候选能正确读回合法中文,但损坏字节也被替换成字符后返回成功;用户看不到读取失败,后续决策继续使用已经改写的事实。

真实子进程证明合法中文读取改善,同时证明 SSH 将损坏正文返回 ok=true,OpenViking 将其作为 completed item;本次可靠读取目标尚未达成。

本次只讨论三个既有子进程文本边界;不新增外部访问、授权、自动安装或语言迁移,也不把 Lark 版本诊断等同业务事实。

规范:spec_ref docs/reference/protocols/manager-evidence-and-continuity-v0.md;spec_revision 4029586;criterion_id “A prompt-only segment receives the declared source read”:失败读取必须显示 unavailable/stale 和 coverage gap,不能冒充当前进展;此 head 的损坏字节反例不满足该失败语义。

改动思路

保留三个现有 provider/transport owner 和显式 UTF-8;业务 stdout 严格解码,诊断输出按需要容错。两个读取 owner 的失败路由不同,不能用一条全局容错规则或新的通用框架代替。

这些 Python 模块承接既有外部 CLI/SSH I/O,归属合理;但业务事实和版本诊断具有不同的完整性约束,必须在各自入口保留该区分。

修复合法 UTF-8 在非 UTF-8 宿主上的解码,同时拒绝损坏的事实/上下文输出;Lark 版本探测可以保留有界诊断容错。

未来相关重构审查:继续使用现有 scanner/provider owner;本次没有需要新增的抽象、并行状态或语言迁移。最近的有效伴随修改是上述最小修复;不把后续猜测当作当前阻塞,也不根据六行或测试数量推断风险。已送达的恢复 RFC 经验不定义这里的验收;只采用普通任务、失败与结果读回的检查方法,不继承历史结论。

具体改动

三处都在既有 text=True 上增加 encoding 与 errors。显式 UTF-8 消除本机默认 code page 对合法协议字节的影响;replace 则在 JSON/schema 检查之前把无效字节变成 U+FFFD。JSON 仍合法,因此原本完整性失败不再进入 unavailable。OpenViking 的同一 _run 还用于读取、写入和不确定写入的核对,不能只按版本输出判断其风险。Lark 的 _version 只返回解析后的版本元数据,并保留有界超时和无本机路径的 public_snapshot,它与完整业务正文的约束不同。

关键代码讲解

  • loopx/capabilities/manager_context/ssh_evidence.py:152 — read_remote:Configured host/grant before and after I/O; decode result -> JSON/schema -> scoped facts, now lossy accepted.

  • loopx/capabilities/context_providers/openviking.py:294 — OpenVikingContextProvider._run:Common real subprocess for preflight,search,read,sync; explicitUTF8 fixeslocale but replace changes bytes.

  • loopx/capabilities/context_providers/openviking.py:382 — OpenVikingContextProvider.retrieve:Validatednamespace/scope -> preflight -> boundedsearch/read -> exacttransient content; corruptbody now completed.

  • loopx/extensions/lark/cli_resolution.py:106 — _version:Five-second diagnostic probe,parsed version only; public snapshot excludes local executable path.

独立验证:Actual real subprocess and production parser return exact Chinese under US-ASCII at H; existing valid UTF8 branch preserved under UTF8locale. Four focused suites pass79tests at bothB/H, including grantrevocation/scope rejection.

负例与完整检查:Identical real child bytes at B/H, CPython3.14.8: valid Chinese under US-ASCII gives B SSH unavailable/OpenViking UnicodeDecodeError, H exact Chinese success. Under UTF-8 malformed byte0xff inside JSON title/content gives B SSH remote_protocol_unavailable with complete=false/no rows and provider UnicodeDecodeError; H SSH ok=true/title m�n and provider completed/content m�n. H does the same under US-ASCII. Real process/decoder/parser exercised; SSH endpoint and provider service synthetic.

运行来源为各自不可变基线/候选 worktree,CPython 3.14.8,验证导入来自该 checkout。Ruff 与 git diff --check 通过;开发期语义 advisory 先于完整检查运行,零候选只说明该探针范围。测试结果:四个聚焦文件,基线和候选各79项通过;两侧完整语义检查通过。没有查询、轮询或等待 GitHub CI。独立探针最初的调用脚本错误已纠正,失败记录保留;它们不是产品失败,也没有用于掩盖上面当前候选的失败。

对主干的风险

Malformed provider/remote bytes remain syntactically valid JSON after replacement and are consumed as current facts/content without a decode failure. 用户的普通路径是继续调用原有命令或已授权读取,没有新增确认或重复配置步骤。No live SSH host, paid provider or Lark account was contacted; only real local children and actual LoopX parsing owners. Native Windows code page not executed; actual US-ASCII default locale demonstrates dependency removal. Sync/reconciliation callers share _run; lossy business-content acceptance blocks this exact head. 上述测试没有扩大权限,合成注册表、临时配置与子进程均不修改活动 Goal 或真实外部服务。正例不能抵消候选的关键失败。既有 schema、quota、scheduler 与授权边界未增加;评审结论也不授予合并权限。恢复必须由原有内容/读取 owner 完成,纠正后重新读回,不能把收到成功字段当作完整性证明。

我的整体评价

REQUEST_CHANGES。ExplicitUTF8 fixes locale handling, but unconditional replacement widens protocol acceptance and falsifies successful readback; concrete current regression. Keep encoding=utf-8 for structured stdout, reject malformed bytes and route decode failure through existing unavailable/contract-invalid paths. Decode only human diagnostic stderr/version text tolerantly where appropriate. Add real-byte positive and malformed-UTF8 negative tests; do not filter all literal U+FFFD because it can be valid source text. 修改范围本身合理,原有 owner 可承接;当前阻塞都有基线/候选差异和可重跑证据。修复后应在新精确 head 重做相关失败检查,并保留本次已通过的合法路径与反方向负例,不能仅改评审标签或削弱检查。

English verdict: REQUEST_CHANGES. Explicit UTF-8 correctly fixes valid Chinese output under a non-UTF-8 parent locale, but errors=replace silently changes malformed structured stdout into accepted evidence/content. A real child writing invalid byte0xff inside a JSON title makes the base report remote_protocol_unavailable with incomplete coverage; this head returns ok=true with m�n. OpenViking similarly returns completed corrupted content. Keep strict UTF-8 for business stdout, handle decode failures through the existing failure owners, and reserve tolerant decoding for bounded human diagnostics. Add actual-byte positive and malformed-input regression tests; valid literal replacement characters must not be blanket-rejected.

Validation: Actual real subprocess and production parser return exact Chinese under US-ASCII at H; existing valid UTF8 branch preserved under UTF8locale. Four focused suites pass79tests at bothB/H, including grantrevocation/scope rejection. Identical real child bytes at B/H, CPython3.14.8: valid Chinese under US-ASCII gives B SSH unavailable/OpenViking UnicodeDecodeError, H exact Chinese success. Under UTF-8 malformed byte0xff inside JSON title/content gives B SSH remote_protocol_unavailable with complete=false/no rows and provider UnicodeDecodeError; H SSH ok=true/title m�n and provider completed/content m�n. H does the same under US-ASCII. Real process/decoder/parser exercised; SSH endpoint and provider service synthetic. No live SSH host, paid provider or Lark account was contacted; only real local children and actual LoopX parsing owners. Native Windows code page not executed; actual US-ASCII default locale demonstrates dependency removal. Sync/reconciliation callers share _run; lossy business-content acceptance blocks this exact head.

Set encoding='utf-8' and errors='replace' on subprocess text-mode
invocations in openviking, ssh_evidence, and lark cli_resolution to
prevent locale code page misdecoding or UnicodeDecodeError on Windows.

Signed-off-by: mika <spacexstarship01@outlook.com>
- Remove silent byte replacement on structured stdout in SSH evidence and OpenViking
- Preserve tolerant replace decoding for diagnostic CLI version probe
- Route decode failures through existing unavailable/protocol failure paths
- Add test coverage for valid UTF-8 Chinese stdout and malformed UTF-8 rejection

Signed-off-by: mikamikasuki <211269698+mikamikasuki@users.noreply.github.com>
@mikamikasuki
mikamikasuki force-pushed the codex/subprocess-utf8-encoding branch from cd089e4 to 7a1c1eb Compare October 21, 2026 06:52
@mikamikasuki
mikamikasuki requested a review from songoow as a code owner October 21, 2026 06:52

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.1-sol · OpenAI · runtime_reported · reasoning_effort=max

APPROVE。旧评审指出的“损坏业务字节被改写成成功事实”已在当前精确 head 修复,合法 UTF-8 的原始目标也通过真实消费者验证。未发现阻塞问题。

动机

在非 UTF-8 系统中读取远程项目事实、检索已配置上下文,或探测 Lark CLI 版本的用户。
同一个子进程已经输出正确的 UTF-8 中文,旧入口仍按本机字符集解码,导致读取失败或版本探测抛异常;用户得不到原本可用的内容。
当前提交能按原文读回合法中文,同时拒绝损坏的业务输出;只有人类诊断用的版本文本允许替换损坏字节。
本次仅修复三个既有文本边界,不新增远程访问、账号权限、自动安装、状态协议或用户配置步骤。

改动思路

业务 stdout 的原文完整性和人类版本诊断的容错具有不同要求;在原有 SSH、OpenViking 与 Lark owner 内分别处理,比引入全局宽松解码器或新框架更小且更准确。
当前 PR 交付显式 UTF-8 的有效内容读取、损坏业务输出的既有失败路由,以及版本诊断的有界容错;不扩展成 provider、持久状态或整个平台迁移。

先让真实子进程产生字节,再由生产读取函数和后续消费者作判断;不能用已解码字符串或直接抛异常的替身证明编码有效。当前 SSH 仍检查授权和返回结构,OpenViking 的读取错误由现有上下文消费者转成 unavailable,Lark 只提取版本元数据。关闭上下文 provider 时仍返回原来的无检索结果,没有自动启用外部服务或增加确认轮次。

输入 当前业务读取 当前版本诊断
非 UTF-8 宿主上的合法中文 原文成功读回 正常取得版本
JSON 内容含非法字节 unavailable,不返回损坏正文 诊断可容错后提取版本
合法文字本来包含 U+FFFD 保留该合法字符 正常解析

表格对应本次真实字节探针,不代表原生 Windows 或在线服务验收。

具体改动

关键代码讲解

完整复审 head 7a1c1eb105e7da2a2490ca5de06d936af9ccb637,实际合并基线 5706ef0552d5c22af805f264ae891556a0f819cf。全部差异为五文件 +77/-0:三个生产调用增加四行参数,另两文件加入三个回归测试。相对旧评审 cd089e4 的修复与当前整头分开检查;基线、调用方和当前测试都重新核实,没有继承旧结论。

  1. read_remote 在原子进程调用明确 UTF-8,保持严格解码;UnicodeDecodeError 属于既有 ValueError 路由,因此输出损坏时返回协议不可用,授权改变后也不会泄露读到的行。
  2. OpenVikingContextProvider._run 为已有版本、搜索、读取与同步传输指定同一编码,移除了旧头的 errors=replace。collect_context_recall 的真实调用证实:损坏正文不会成为 completed 内容,而会进入现有失败分支。
  3. _version 保留五秒上限和纯版本读回,仅诊断文本采用替换容错;可执行文件发现与账户授权规则没有改变。

规范依据:spec_ref docs/reference/protocols/manager-evidence-and-continuity-v0.md,spec_revision 5706ef0552d5c22af805f264ae891556a0f819cf,criterion_id A prompt-only segment receives the declared source read。该原有条款要求失败来源保持 coverage gap,不冒充当前进展,也不拖垮整个 Turn。本头通过既有 SSH 不可用结果及真实上下文消费者的 unavailable 读回满足它;普通编码修复和诊断范围还沿原 PR 请求及旧评审判定,没有新增全仓义务。

对主干的风险

我在 CPython 3.12.15 的 US-ASCII 和 UTF-8 父进程环境中,使用相同真实子进程字节检查 ASCII、中文、合法 U+FFFD、非法 0xff、撤销、越界来源与关闭路径:当前两种环境各15项均满足预期,基线分别8/15与14/15。另补同一主机的未授权 Goal、新增未授权结果及拒绝后恢复的实际读取,两侧都保持范围边界。没有连接真实 SSH、云端 OpenViking 或 Lark 账户,也未执行原生 Windows code page。

四个聚焦文件当前90项、基线87项通过;source canary premerge 的6项直接检查和16项选中检查通过,DCO与差异检查通过,没有查询或等待CI。新加的部分作者测试用替身输出字符串或直接抛异常,不能单独证明真实解码;本次独立字节验证补足了这点。可将其中一个现有测试换成真实字节回归,属于非阻塞的覆盖改善,不需要新抽象或新测试框架。没有新增状态词表、prose 分类规则或自动加载指令。

我的整体评价

APPROVE。long_horizon 与 user_experience 在这条既有读取链上改善:合法内容可继续使用,损坏内容保持明确失败,恢复后原入口继续返回合法结果;没有增加人工准备或确认。改动复用原 owner,生产成本与缺陷相称,也没有以编码修复引入另一套兼容协议。该判断只覆盖当前源码和本次真实本地传输验证,不宣称安装态、多平台或长期净收益,也不授予 dismissal 或 merge 权限。

English verdict: APPROVE — head 7a1c1eb. Explicit UTF-8 fixes real non-UTF-8-parent failures while structured SSH/context output remains strict; malformed bytes become unavailable and valid literal replacement characters survive. Only bounded version diagnostics are tolerant. Real local-child consumer and scope/recovery controls,90 focused head tests and the native risk gate passed; native Windows and live services remain unqualified.

@huangruiteng
huangruiteng merged commit f810b92 into loopx-project:main Oct 21, 2026
28 of 40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants