Repository navigation
feat(eval): structured write allowances for the group, sweep and file skills (#67, step 3 of 3) - #100
Merged
Conversation
… skills (#67) The four group skills, reconcile-github-issues, file-github-issues, explore-github-repo and setup-github-agent-workflow list what they may write as allow entries, which completes ADR 0017. A create_issue or create_pr allow now has a required labels list, so new issues with no labels and a PR labelled agent:needs-work are checked.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 3 of 3 for #67, on top of #97 and #99 (both merged). The last contracts move to
allowentries, so every contract that writes is now checked by what it writes and not only by where.Migrated
triage-github-issues,implement-github-issues,review-github-prs,shepherd-github-prs): one unmarked run-log comment each.reconcile-github-issues,file-github-issues,explore-github-repo.setup-github-agent-workflow, which wasn't in the plan but maps directly ontolabel_create,label_updateandwrite_file.After this, no contract that writes still lists its writes as prose.
One schema addition:
labelson createsStep 1 left
create_issueandcreate_prmatched by op alone, so "new issues, with no labels" couldn't be enforced. Anallowfor either now needslabels: the labels the new item may carry, each a name or glob. Every label it is made with must match one, and[]means none. It is required (not optional) so an author has to decide, and the schema rejects it on every other op.file-github-issues,explore-github-repo:{ op: create_issue, labels: [] }implement-github-issue(tightened from step 2):create_prwithlabels: [agent:needs-work],create_issuewithlabels: []. Its goldens already matched.Decisions worth a look
reconcile-github-issuesmay add any one of the four state labels and remove the others. Its repair picks the target by a priority order fromlabels.mdthat the checker doesn't model, so the list permits the whole mutex set, and the sentence says so. It never listsneeds-triage, a lease label to add, or a lease marker.allowis declarative for now. Scenarios already forbid a group from doing item work with an earlier, stricter rule, and the group goldens have no effects. So thesharedentry is covered by direct contract tests and, later, by real runs througheval.Tests
13 new tests. Schema:
labelsrequired on creates, rejected elsewhere. Matcher:allowedBywithlabels, including[], globs and every-label-must-match. Forbidden-effect rows against the real goldens (filing never labels an existing issue, reconcile never applieswontfixor writes a lease, setup never pushes), and a check that each group's one write is an unmarked run-log comment. With the allow check disabled, 26 scenario tests fail.npm test: 561/561 (548 + 13).npm run validateandcheck-plugin-versionpass. ADR 0017 is updated to say all three steps are built, and there is a changeset.Known limits (also in the ADR)
closeandreview_requestare matched by op alone, and a created item's other fields (title, body, parent, PR base) are not constrained.Closes #67. Refs #72.