FE-1232: Report comparison evidence

[kostandinang]

Why

Comparison findings need a reusable publication path that does not blur Notion editing with validity, contamination, test design, judging, and redaction policy.

What

Add two separate, composable project skills:

• /notion-reporting — triggers when a user asks to create or revise a Notion report, summarize findings on a Notion page, or add an overview or side note. It discovers the live Notion MCP surface, fetches before editing, applies the smallest safe mutation, preserves child content, and fetches again to verify the result.
• /comparison-reporting — triggers when designing or reporting elicitation, execution, or end-to-end comparison campaigns. It freezes test axes, rubrics, judge protocol, and repeat contracts; evaluates validity before outcomes; separates procedure determinism from output similarity; classifies implementation/runtime/protocol issues; and builds requirement-level elicitation-to-implementation traceability.

The comparison skill also enforces masked/unblinded/controller-only evidence boundaries and composes with /notion-reporting only for safe Notion formatting and mutation.

A dependency-free contract check pins skill discovery metadata, direct references, mutation safety, campaign-design guardrails, invalid-attempt retention, controller-only redaction, and the rule against unsupported winner or determinism claims.

Example use:

/comparison-reporting
Design a greenfield whole-app campaign with a frozen rubric and masked judge protocol, then report the resulting elicitation, execution, determinism, and end-to-end evidence to <Notion URL>.

/notion-reporting
Update <Notion URL> with an overview, problem, result, findings, limitations, and side note from this evidence.

How to test

Automated

1. npm run check:skills
   • Expected: the existing 21 ln-* contracts and both reporting-skill contracts pass.
2. npx oxlint scripts/check-reporting-skills.mjs
3. npx oxfmt --check scripts/check-reporting-skills.mjs
4. npx remark .agents/skills/comparison-reporting/SKILL.md .agents/skills/comparison-reporting/references/*.md --quiet --frail
5. npm run check:promoted-run-paths
6. npm run build
7. Use the PR Full gate as the authoritative test result.

Local npm test currently reaches 2,490 passing tests before the unrelated standalone-web-session-host.tui-differential.test.ts baseline race fails with “Agent is already processing”; the focused rerun reproduces it without any reporting-skill code involved.

Manual skill smoke

1. Start a new agent chat so project skills are rediscovered.
2. Invoke /comparison-reporting for a planned campaign and confirm it freezes codebase mode, change scope, interface type, plan stability, sample/repeat count, rubric, winner rule, and judge protocol before outcomes.
3. Supply repeated execution attempts and confirm the report separates canonical transition/action sequence evidence from generated-output structural similarity and bounds the determinism claim by valid attempt count.
4. Supply a matched elicitation-to-execution case and confirm it produces requirement rows for explicitly elicited, inferred, implemented, missed, and verified outcomes without making a causal claim from one pair.
5. Publish through /notion-reporting and confirm fetch-before-edit, one targeted mutation, preserved child pages/databases, and post-write verification.

[kostandinang]

• FE-1232: Report comparison evidence #346 👈 (View in Graphite)
• FE-1230: Compare greenfield execution #345
• next

This stack of pull requests is managed by Graphite. Learn more about stacking.

[cursor]

PR Summary

Low Risk
Documentation and agent-skill guidance only, with no product runtime or schema changes; risk is limited to incorrect external Notion edits if agents misapply the skills.

Overview
Introduces two project-local agent skills for safe Notion publication and validity-first comparison reporting, with reference docs for elicitation, execution, end-to-end traceability, frozen campaign strategy, and example report wording.

/notion-reporting covers MCP discovery, fetch-before-edit, smallest safe mutations (preserving child pages/databases), and post-write verification. /comparison-reporting stays separate: it classifies comparison kind, orders evidence (validity before outcomes), enforces masked/unblinded/controller-only boundaries, blocks unsupported winners and causal overreach, and composes with Notion reporting only for formatting—not for evidence policy. Active procedures like /compare-specs are explicitly not retrofitted with rigorous campaign machinery.

scripts/check-reporting-skills.mjs pins safety-critical phrases, validates skill metadata, forbids exposing the execution case root in public evidence, and recursively checks relative Markdown links. npm run check:skills now runs this after the existing ln-* check (renamed log labels in check-ln-skills.mjs). A patch changeset and memory/PLAN.md record FE-1232 completion.

^{Reviewed by Cursor Bugbot for commit fdd581f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ae960e3. Configure here.}