Skip to content

feat(eval): structured write allowances for the group, sweep and file skills (#67, step 3 of 3) - #100

Merged
glennfaison merged 1 commit into
mainfrom
feat/structured-effects-group-skills
Nov 14, 2026
Merged

glennfaison merged 1 commit into
mainfrom
feat/structured-effects-group-skills

Conversation

@glennfaison

Copy link
Copy Markdown
Owner

Step 3 of 3 for #67, on top of #97 and #99 (both merged). The last contracts move to allow entries, so every contract that writes is now checked by what it writes and not only by where.

Migrated

  • The four group skills (triage-github-issues, implement-github-issues, review-github-prs, shepherd-github-prs): one unmarked run-log comment each.
  • reconcile-github-issues, file-github-issues, explore-github-repo.
  • setup-github-agent-workflow, which wasn't in the plan but maps directly onto label_create, label_update and write_file.

After this, no contract that writes still lists its writes as prose.

One schema addition: labels on creates

Step 1 left create_issue and create_pr matched by op alone, so "new issues, with no labels" couldn't be enforced. An allow for either now needs labels: the labels the new item may carry, each a name or glob. Every label it is made with must match one, and [] means none. It is required (not optional) so an author has to decide, and the schema rejects it on every other op.

  • file-github-issues, explore-github-repo: { op: create_issue, labels: [] }
  • implement-github-issue (tightened from step 2): create_pr with labels: [agent:needs-work], create_issue with labels: []. Its goldens already matched.

Decisions worth a look

  • reconcile-github-issues may add any one of the four state labels and remove the others. Its repair picks the target by a priority order from labels.md that the checker doesn't model, so the list permits the whole mutex set, and the sentence says so. It never lists needs-triage, a lease label to add, or a lease marker.
  • A group's allow is declarative for now. Scenarios already forbid a group from doing item work with an earlier, stricter rule, and the group goldens have no effects. So the shared entry is covered by direct contract tests and, later, by real runs through eval.

Tests

13 new tests. Schema: labels required on creates, rejected elsewhere. Matcher: allowedBy with labels, including [], globs and every-label-must-match. Forbidden-effect rows against the real goldens (filing never labels an existing issue, reconcile never applies wontfix or writes a lease, setup never pushes), and a check that each group's one write is an unmarked run-log comment. With the allow check disabled, 26 scenario tests fail.

npm test: 561/561 (548 + 13). npm run validate and check-plugin-version pass. ADR 0017 is updated to say all three steps are built, and there is a changeset.

Known limits (also in the ADR)

close and review_request are matched by op alone, and a created item's other fields (title, body, parent, PR base) are not constrained.

Closes #67. Refs #72.

… skills (#67)

The four group skills, reconcile-github-issues, file-github-issues,
explore-github-repo and setup-github-agent-workflow list what they may
write as allow entries, which completes ADR 0017. A create_issue or
create_pr allow now has a required labels list, so new issues with no
labels and a PR labelled agent:needs-work are checked.
@glennfaison
glennfaison merged commit 875c84a into main Nov 14, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refactor the effects.writes entries to be checkable, not just prose

2 participants