Jev 可选观察器：按指定文件变化评估任务进展，接入已有重规划义务——功能、效果与裁判对照

[songoow]

当前结论：指定文件变化采集、独立评估与接入已有重规划义务已实现，可靠纠偏尚未证明

当前实施与评审入口是 PR #4854（head 21922b43e），目标分支为 main。它包含基于实际产物的任务进展观察，以及一个默认关闭的核心 capability progress_review（off / shadow / assist）：assist 下，连续的类型化漂移回执触发已有的 autonomous_replan_obligation，不新增任何权限。Jev 是可选观察器，核心不导入它、不调用模型。PR 尚未合并。

本帖已按当前范围重写：此前的多方向排序比较和“只保留非主分支”的建议不再作为当前实施结论。讨论演进保留在中文决策记录 / English，这里说明当前功能、实际效果和未完成项。

已经做到： 在显式接入的刷新点读取真实前后产物，由独立 Jev 观察器评估（三道 Noul＋两道 Choice，都问检查点之间的变化），写入类型化回执；核心从判断重算漂移布尔值、按 turn 身份关联 run；assist 下连续漂移回执触发已有重规划义务，Agent 只能用类型化证据解除（带整个义务窗口之外新证据的新假设、真实 blocker、终态覆盖或证据关联 vision path），改名不算、回放窗口内已做过的声明也不算；已形成的义务不因更新的未评估回执消失；回执按 run 时间排序，不跨观察器比较本地 sequence。支持 Goal 策略与 pin、去重、预算、失败记录和分阶段计时。
还没做到： 在真实长期 Goal 上证明可靠发现跑偏、Agent 确实换方向、提高最终完成率或减少无效工作。不触发暂停、user gate 或验收判定。

1. 具体实现了什么

功能	当前实际行为	达到的效果与边界
初始化观察范围	drift init 绑定 Goal 契约、精确文件、初始基线	不依赖 Agent 自写“我做了什么”作为唯一证据；契约和文件范围仍要由操作者指定
真实变化采集	drift refresh 包装已有核心刷新命令，读取前后限定文件及 delta	在已包装的调用点自动采集，关联真实 run；并非自动安装到所有原生会话
独立评估	drift drain 在另一个进程判断“目标关系”和“证据增量”	模型调用不进入核心事务；这是额外观察，不接管执行 Agent
去重与恢复	持久化事件身份、请求预留和结果；检查配置/契约变更	重复事件不会重复计数，已保存响应可在重启后复用；发送结果不明时不自动重复请求
接入核心	回执写入 Goal 运行时；核心 capability progress_review 按严格 schema 读取、重算布尔值、关联 run；assist 把连续漂移回执变成已有 autonomous_replan_obligation	模型无权限，义务来自 Goal 负责人的策略；解除只能靠类型化证据，改名或重提旧声明被拒；未评估回执不形成也不解除义务
可配置与回退	观察器本地 off/shadow、出站开关、请求预算；Goal 策略 off/shadow/assist、信号、阈值与 pin 走 configure-goal / Dashboard / chat 配置 API；key 只走 TYPESAFE_API_KEY	有 key 不自动启用功能；缺 key、认证失败、超时不改变原 Agent 流程，也不会自动启动备用裁判；pin 是手动的，basis 变化时 status 提示 rebind_hint
结果读回	drift status 展示判断、未知、失败和耗时	有可复核记录，但记录不等于验收通过；状态输出不打印私人源码

完整操作、关闭与卸载见中文操作指南 / English。这次不包含 D2–D8、不修改核心选择器、没有原生 hook；配置入口有 configure-goal、Dashboard capability 编辑器和 chat 配置 API，没有 Lark 入口。

观察范围如何选择

初始化时用 --path 指定精确文件，例如 src/retry.py、tests/test_retry.py、reports/retry_probe.json。观察器读取清单中文件的前后内容及变化，另外使用 basis 中的目标、验收条件和显式证据引用，不自动扫描全仓或读取所有 Agent 上下文。最多 32 个选定文件，并有字节上限；清单外材料可能影响判断，不能假装不存在。

这限制的是观察器输入，不是 Agent 的编辑权限。Agent 可以在其他文件推进工作，因此 no_delta 不等于整个任务没有进展。读取测试报告也不意味着观察器执行了测试。当前实际便利是自动整理这些材料、保存可追溯判断；节省多少人工时间尚未测量。

2. 当前实现的录制对照结果

对照 harness 回放冻结的 16 序列矩阵（9 个漂移序列：6 个从第 1 轮起纯装饰性、3 个先做真实工作后才漂移；7 个标注 on-goal 的真实上游提交；每轮自报 advanced 并更换 hypothesis id）。已提交的 live 录制使用问题集 scoped-progress-sentinel-v2、jev-1.13.0、标签阈值 0.6，35 轮，34 轮 completed，1 轮按失败关闭。回放不需要 key，CI 可复现同一组数字。

	类型化重复保险丝	noul 回执	choice 回执
在 gold 轮被标记的漂移序列	0/9，按构造不可见	9/9	5/9
阈值 2 下达到 assist 义务	0/9	9/9	2/9
真实 on-goal 提交被误报	0/7	已评估的 0/6	已评估的 0/6
真实 on-goal 提交无判定（按失败关闭）	—	1/7	1/7
混合序列内的提前告警	0	0	0

纯装饰性序列第 1 轮被标记、第 2 轮即可触发义务；混合序列恰在各自漂移轮（第 3 轮）被标记：实现落地后的装饰性改动 serves_acceptance 为 0.06–0.13，旧措辞下是 0.62–0.90。负结果探测（evidence_increment 0.85）与必要失败测试（serves_acceptance 0.73）不被标记。第二次独立 live 复现了全部 16 个序列的结果。客户端评估延迟中位 1.45–1.49 s、P95 2.9 s（早先录制 0.74–0.81 s，差异来自网络与 provider 时段），输入 token 中位 1890。

必须保留的限定： v2 措辞是在看到 v1 漏检这些构造序列之后修订的，构造用例对新措辞不再是留出集；只有 7 个单轮真实提交未被调参触碰。按失败关闭的那一轮是真实 on-goal 提交 fix_manager_refused_read_argument，它没有判定，既不是误报也不是确认通过，写成“0/7”会夸大证据。此前本帖记录的两道 Choice、5 场景×2 次、4/10 标签匹配的重测属于更早的实现，已被上述结果取代。

3. 工程验证与效果验证分开看

• 工程验证（head 21922b43e）： 77 项包内测试、80 项引用该能力或共享出口策略的核心测试、46 项 replan-ack/refresh-gate、780 项 ratchet/架构边界、7 项 TS replan_semantics；kernel mypy（CI 配置）、新模块 strict mypy、typecheck:control-plane、Ruff、文档治理/仓库卫生/sentinel 回放 smoke 通过。真实 refresh-state 闭环两条序列：义务触发、loopx status 显示、改名被拒、pending→failed 之上义务保持、类型化 ack 与 continue vision path 解除并重新武装。
• 效果证据： 冻结矩阵只能说明当前输入/题型/模型组合在这些序列上的表现，且不是留出集。尚无真实长期 Goal 的 shadow 数据、完整 Agent 对照、相对周期复审的提前量或无效工作减少量。
• 权限边界： 不改 Goal/Todo/验收/结算权威；不把高概率变成自动操作权限。assist 会改变工作契约（产生 required 义务与 stop_condition），但模型不获得任何权限，义务由 Goal 负责人的策略产生、由已有 writeback 契约解除。L1 reliability-diagnostics 的无出站契约与本模型评估保持独立。

4. 为什么止于接入已有义务，以及下一步讨论什么

讨论从“规则保险丝有没有盲区”推进到“能否用独立证据更早发现问题”。其中已经修正两点：单个重复触发器漏检，不等于整个 LoopX 看不见；少量案例正确，不证明 Jev 是规则的严格超集。Jev 和独立 Agent 都可以承担额外判断，材料准备与问题定义必须和模型选择分开评价。

本次提交可回退的观察工具和一个默认关闭的核心 capability，复用已有重规划义务而不另建监督器，先让材料采集、结果核查和 ack 契约有真实入口。是否收录这个可选工具，由 PR 评审决定；RFC #4749 的讨论稿收录不自动批准模型采用或干预。

接下来值得明确的是：

1. 如何独立定义“目标相关”“产物变化”“新增验证证据”，避免混成一个分数？
2. 补齐可归属验收材料后，是否能稳定检出装饰性工作，同时不误伤必要测试、研究和合理等待？
3. 相比原流程及独立 Agent 裁判，Jev 是否减少了复核成本或无效工作？

在这些问题有证据前：默认 off；真实 Goal 先开 shadow 并用 drift label 标注两周；assist 只在单个 pin 过的 Goal 上试并读取 Agent 的 ack 行为；不加暂停/升级，不降低阈值追求更好看的命中率。

5. Jev、Codex 与 Claude 的裁判方法对照

这部分补充的是同一组十份小 diff 的判断器实验，Claude 是对原 Jev/Codex 对照的补测，不是第二组独立样例，也不是当前观察命令的效果数据。

裁判	最终告警匹配预期	客户端中位耗时	CLI API 时长中位数	每例平均报告/估算成本
Jev，三道 Noul	10/10	645 ms，HTTP 区间	—	约 $0.00003，历史估算
Codex	10/10	9.236 s，CLI wall	—	未报告
Claude Haiku	10/10	10.187 s，CLI wall	9.191 s	$0.00844，CLI 报告
Claude Sonnet	10/10	5.402 s，CLI wall	4.191 s	$0.00271，CLI 报告
Claude Opus	10/10	8.054 s，CLI wall	7.019 s	$0.00528，CLI 报告

已重新核对保存结果和脚本，完整模型标识、计时/成本口径和限制见决策记录的“裁判方法对照”。主要修正如下：

• 10/10 是最终告警匹配，不是所有子判断的准确率。Jev 用“行为变化概率低于 0.5 或 服务验收概率低于 0.5”告警，Agent 用对应两个布尔值。
• 第三道 Noul 实际是核对自报摘要 summary_supported，没有参与上述计分；不是当前实现的证据增量问题。输入也确实带了统一自报摘要和 tests_pass，不能说完全排除了自述。
• Claude 首轮单 turn 上限截断了 Haiku 6 条、Opus 3 条结构化输出；这些是无效调用，不能算模型判错。表格采用统一允许三轮的有效结果。
• 同一设计者的十个小样例（424–1095 字节），各单次运行；Codex 在较早时段运行。Jev HTTP 耗时与完整 CLI wall 口径不同，不支持纯推理加速倍数；成本也不是同口径实付账单。
• 不能据此得出“只有 Jev 能做频繁检查”“Sonnet 是最佳裁判”或“Agent 布尔值不能接保险丝”。这组结果只支持继续测试不同评估方法。

当前观察器同时记录三道 Noul 与两道 Choice，但生产规则 progress_review_signal_rule_v1 以 serves_acceptance ∧ evidence_increment 为门槛，不是这里的 behavior_change 规则（那条规则会误伤没有运行时行为变化的有效测试、文档和前置工作），也没有接入 Claude/Codex 升级复核。这组裁判对照只支持继续在同一批材料和独立标签上比较评估方法。

English summary: PR #4854 (head 21922b43e) targets upstream main. An optional Jev observer captures scoped before/after file state at explicitly wrapped refresh-state calls, evaluates the change between checkpoints (3 Noul + 2 Choice) out of process, and writes typed receipts; a default-off core capability progress_review recomputes the drift booleans, joins receipts to runs, and under assist raises the existing autonomous_replan_obligation, discharged only by typed evidence (new identifiers with evidence ids absent from the whole obligation window, a concrete blocker, terminal coverage, or an evidence-linked vision path); replaying a claim already made in the window is refused. On the committed v2 recording the noul signal flags 9/9 drift sequences and reaches the obligation on all nine, with 0/6 false flags on the real commits that completed evaluation (one failed closed, no verdict). The matrix is not held-out for the v2 wording; no real-Goal shadow data, intervention study or task-benefit evidence exists yet. This discussion supersedes the earlier multi-direction, non-main and two-Choice summaries.

[songoow]

Update: the progress-review loop now closes inside LoopX, and the differential is reproducible in the repo. Draft PR #4854 has been extended on the same branch.

What changed:

• A default-off core capability progress-review-sentinel reads typed receipts from the optional observer. shadow records and shows them in loopx status; assist turns consecutive completed drift receipts into the existing autonomous_replan_obligation. An acknowledged replan re-arms it. No pause, gate or acceptance authority is added.
• The observer asks three Noul questions next to the two Choice questions and writes typed drift signals, never probabilities the core must interpret.
• loopx-jev sentinel compare replays a frozen 16-sequence matrix against committed recordings, so the numbers below reproduce in CI without a key.
• One end-to-end test runs the same real refresh-state sequence through off, shadow and assist: only assist raises the obligation, and a real acknowledged replan re-arms it.

Committed live recording (jev-1.13.0, 35 rounds):

	Typed repeat fuse	noul	choice
Drift sequences flagged (9)	0	6	4
Reaching the assist obligation	0	6	1
Real on-goal commits falsely flagged (7)	0	0	0

All six purely cosmetic sequences were flagged at round 1 and would raise the obligation at round 2; the periodic review waits for 20 durable runs. Median assessment latency 807 ms, p95 1.5 s, 1879 median input tokens.

What it does not do: cosmetic churn after a real implementation landed in the same file was not flagged (serves_acceptance stayed 0.62–0.90); one executed negative probe was flagged in two of three live runs; run-to-run noul detection varied 6–7 of 9; one of 35 answers in one run failed the strict decoder and closed as failed. Constructed gold labels come from their author. No claim of reduced wasted work; that needs an intervention study.

Details, boundaries and the full table: PR #4854 and loopx/capabilities/progress_review/README.md.

[songoow]

Correction and update (2026-09-22). An external review of the first closed-loop head found that the comparison harness passed a case-derived goal_id into the model input, that the noul rule gated on behaviour change rather than goal contribution, and that receipts were not bound to the current goal contract or checked for Agent/Todo identity. Please read the earlier numbers in this thread (noul 6/9) as superseded.

PR #4854 head a49a11295 fixes all four deterministically: whitelisted model input with a byte-identity test, rule v1 gating on serves_acceptance and evidence_increment about the change between checkpoints (core recomputes and rejects inconsistent receipts), a required pinned contract_revision for assist, and identity/pending rules for the streak.

Re-recorded v2 differential, reproduced by a second independent live run on all 16 sequences: typed fuse 0/9, noul 9/9 flagged at the gold round and 9/9 reaching the obligation, 0/7 false flags on real upstream commits, 0 premature; choice 5/9. Caveat kept in the docs: the v2 wording was revised after seeing the v1 misses on the constructed sequences, so those are no longer held-out for the wording; the real commits were untouched. assist changes the work contract (a required obligation) even though it grants no new authority; still Draft, no claim of reduced wasted work.

[loopx-agent]

Implementation update: progress judgments, uncertainty and versioned replay

Thanks to songoow for the optional-observer → typed-receipt → existing-replan integration in #4854. It merged on September 23. The opening post preserves an earlier, pre-merge checkpoint; the following describes source revision 3203b558, rather than a verified installation on every host.

The useful distinction is whether the change serves acceptance and whether it adds evidence, rather than simply whether runtime behavior changed. A useful negative finding can add evidence without changing behavior. Conversely, changing behavior does not establish contribution to the Goal.

The core recomputes the receipt's signal from its typed judgments and rejects inconsistent booleans. For noul, these are illustrative inputs, not observed model scores:

Threshold	Serves acceptance	Evidence increment	Signal
0.6	0.2	0.3	Drift
0.6	0.2	0.8	Not drift
0.6	0.5	0.5	Undecided (null)
0.5	0.5	0.2	Undecided under signal-rule-v2; drift under historical v1

That last row is a version boundary: v2 preserves uncertainty when the classes overlap; historical v1 receipts retain their original tie rule. The scoped-progress-sentinel-v2 question set and progress_review_signal_rule_v2 signal rule are separate version axes. Upgrade readers before writing v2 receipts into a shared runtime. Exact rule and validation.

Unknown/failed evaluations cannot form a new drift streak, and do not by themselves erase an already formed obligation. off reads nothing; shadow observes; pinned assist can require the existing replan acknowledgement. “Not drift” is not task acceptance, and an observer score grants no execution authority. For canonical task criteria, the current acceptance_scope route reads the acceptance owner; a manually supplied basis remains an operator study. Criterion binding and readback.

I checked the table, historical tie behavior, failed/unknown signals and malformed-receipt rejection against the unchanged receipt module in an isolated namespace, without a model call or live Goal. This checks those mechanics. The recorded comparison above still has its stated non-held-out limitation; it does not establish reliable correction or reduced wasted work on real long-running Goals.

中文补充： songoow 的#4854已于9月23日合入；主帖中的“未合并”是历史检查点。当前规则把“服务验收”和“新增证据”分开判断，不用运行时行为变化替代目标贡献。规则v2在分类重叠处保留未知，旧v1回执仍按原规则回读；问题集v2与信号规则v2不是同一个版本号。未知不等于通过，也不能自动消除已经形成的重规划义务。以上核对的是固定源码中的规则与拒绝路径，没有开启观察器、调用模型或运行真实Goal，也没有新增长程效果结论。