Repository navigation
feat(work-items): triage reads the glossary, ADRs and scope statement first; outside-PR intake off by default #11908
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
kyle-sexton
merged 5 commits into
main
from
work-resolve-issue/issue-11867-41281563218-1
Nov 1, 2026
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
df5ce8d
Resolve issue #11867
melodic-automation-lanes[bot] a84c06b
Merge main into work-resolve-issue/issue-11867-41281563218-1
melodic-automation-lanes[bot] 2d7029a
Address review feedback on work-resolve-issue/issue-11867-41281563218-1
melodic-automation-lanes[bot] c55d60a
Address review feedback on work-resolve-issue/issue-11867-41281563218-1
melodic-automation-lanes[bot] 0bff85a
Address review feedback on work-resolve-issue/issue-11867-41281563218-1
melodic-automation-lanes[bot] File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
12 changes: 12 additions & 0 deletions
12
.changes/work-items/work-resolve-issue-issue-11867-5d2e8c71.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| --- | ||
| bump: minor | ||
| --- | ||
|
|
||
| ### Added | ||
|
|
||
| - `/work-items:triage` reads the repository's glossary, the ADRs covering the item's area and its scope statement before it evaluates an item. It names which of them it found in a `Read first:` line (#11867). | ||
| - A `triage_outside_prs` repository setting in `docs/conventions/work-items.yaml`, default `false`, says whether the repository takes requests as pull requests from outside contributors (#11867). | ||
|
|
||
| ### Changed | ||
|
|
||
| - With `triage_outside_prs` unset or `false`, the `/work-items:triage` attention view lists no pull requests and does not fetch them. A pull request named explicitly is still triaged (#11867). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
25 changes: 25 additions & 0 deletions
25
plugins/work-items/evals/tr-numbered-states-read-first/case.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| schema_version: "1.1" | ||
| name: tr-numbered-states-read-first | ||
| description: 'A numbered triage reads the project''s own record before it evaluates and names what it found: /work-items:triage 31, interactive, with the item given in the prompt. The workspace starts empty, so it holds no glossary, no ADRs and no scope statement, and step 1 must record each as none on a Read first: line rather than skip the line or invent a path. A run needs --allow-tools Bash.' | ||
| tags: [triage, read-first, numbered] | ||
| plugins: ["../.."] | ||
| expected_outcome: 'The run searches the workspace for a glossary, ADRs and a scope statement, finds none, and its triage output carries the line Read first: glossary none, ADRs none, scope none before its recommendation. It presents the recommendation and waits: it runs no tracker write (no label, comment or close).' | ||
| execution: | ||
| prompt: "/work-items:triage 31\n\nInteractive session. The adapter's \"View item\" read for this repository returns issue #31 'CSV export drops the last row', opened 2026-10-02 by a reporter, with no labels, no comments and no linked pull request. Its body: 'Exporting a report with 10 rows to CSV gives a file with 9 data rows. Expected 10.' Use this item as given; there is no other tracker to read. Present your recommendation only and change nothing." | ||
| allowed_tools: [Bash, Read, Glob, Grep] | ||
| max_turns: 15 | ||
| timeout_seconds: 240 | ||
| graders: | ||
| - name: states-read-first-none | ||
| type: regex | ||
| pattern: '^[^\n]*\bRead first\b\W*glossary\W*none\b[^\n]*?\bADRs?\W*none\b[^\n]*?\bscope\W*none\b' | ||
| flags: 'mi' | ||
| target: last_message | ||
| arm: both | ||
| - name: no-tracker-write | ||
| type: tool_used | ||
| tool: Bash | ||
| input_match: '(?:\b|(?<=\\n))gh[ \t]+(?:issue|pr)[ \t]+(?:edit|comment|close)\b|(?:\b|(?<=\\n))gh[ \t]+label[ \t]+(?:create|edit|delete)\b|(?:\b|(?<=\\n))gh[ \t]+api\b(?:(?:(?!\\n|;|&&|\|)[^\n])*?(?:-(?:X|-method)[ \t=]*(?:POST|PATCH|PUT|DELETE)\b|\bmutation\b)|(?!(?:(?!\\n|;|&&|\|)[^\n])*?-(?:X|-method)[ \t=]*GET\b)(?:(?:(?!\\n|;|&&|\|)[^\n])*?--input\b|(?!(?:(?!\\n|;|&&|\|)[^\n])*?[ \t]graphql\b)(?:(?!\\n|;|&&|\|)[^\n])*?(?:[ \t]-[fF]\b|--(?:raw-)?field\b)))' | ||
| min: 0 | ||
| max: 0 | ||
| arm: both | ||
120 changes: 120 additions & 0 deletions
120
plugins/work-items/evals/tr-numbered-states-read-first/samples/no-tracker-write.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,120 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "ls -a docs README.md" | ||
| } | ||
| } | ||
| ], | ||
| "why": "searches the workspace" | ||
| }, | ||
| { | ||
| "answer": [], | ||
| "why": "uses the given item" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "gh api repos/o/r/issues/31\ngrep -F CSV README.md" | ||
| } | ||
| } | ||
| ], | ||
| "why": "reads the item through the API, then searches on its own line" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "gh api -X GET search/issues -f q='repo:o/r is:issue CSV'" | ||
| } | ||
| } | ||
| ], | ||
| "why": "searches through the API with an explicit GET and a field" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "gh api graphql -f query='{ repository(owner:\"o\",name:\"r\"){ issue(number:31){ title labels(first:5){nodes{name}} } } }'" | ||
| } | ||
| } | ||
| ], | ||
| "why": "reads the item through a GraphQL query" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "gh issue edit 31 --add-label \"type: bug\"" | ||
| } | ||
| } | ||
| ], | ||
| "why": "labels the item" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "gh issue comment 31 --body \"Triage: reproduce first\"" | ||
| } | ||
| } | ||
| ], | ||
| "why": "comments on the item" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "gh api -X POST repos/o/r/issues/31/labels -f labels[]=bug" | ||
| } | ||
| } | ||
| ], | ||
| "why": "labels through the API" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "ls docs README.md\ngh issue edit 31 --add-label bug" | ||
| } | ||
| } | ||
| ], | ||
| "why": "labels the item on a later line" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "gh api repos/o/r/issues/31/labels -f 'labels[]=bug'" | ||
| } | ||
| } | ||
| ], | ||
| "why": "labels through the API with an implied POST" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "gh api graphql -f query='mutation { addLabelsToLabelable(input: {labelableId: \"I_1\", labelIds: [\"L_1\"]}) { clientMutationId } }'" | ||
| } | ||
| } | ||
| ], | ||
| "why": "labels through a GraphQL mutation" | ||
| } | ||
| ] | ||
| } |
30 changes: 30 additions & 0 deletions
30
plugins/work-items/evals/tr-numbered-states-read-first/samples/states-read-first-none.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": "Read first: glossary none, ADRs none, scope none", | ||
| "why": "template line" | ||
| }, | ||
| { | ||
| "answer": "**Read first:** glossary `none`, ADRs `none`, scope `none`\n\nRecommendation: Bug, priority: medium, needs reproduction.", | ||
| "why": "bold label and code spans" | ||
| }, | ||
| { | ||
| "answer": "- Read first: glossary: none, ADR: none, scope: none (searched README*, docs/)", | ||
| "why": "colon form with a search note" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": "Read first: glossary docs/glossary.md, ADRs none, scope none", | ||
| "why": "names a glossary the empty workspace does not hold" | ||
| }, | ||
| { | ||
| "answer": "Recommendation: Bug, priority: medium. No glossary, ADRs or scope statement found.", | ||
| "why": "no Read first line" | ||
| }, | ||
| { | ||
| "answer": "Read first: glossary none, ADRs none", | ||
| "why": "scope left out" | ||
| } | ||
| ] | ||
| } |
37 changes: 37 additions & 0 deletions
37
plugins/work-items/evals/tr-outside-prs-unset-lists-no-pr/case.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| schema_version: "1.1" | ||
| name: tr-outside-prs-unset-lists-no-pr | ||
| description: 'Outside-PR intake is off by default: the workspace has no docs/conventions/work-items.yaml, so triage_outside_prs is unset and resolves false. The attention view lists the unlabeled issue #31 and no pull request, though the listing in the prompt also carries an unlabeled outside-contributor PR #32. A run needs --allow-tools Bash.' | ||
| tags: [triage, attention-view, outside-prs, must-not] | ||
| plugins: ["../.."] | ||
| expected_outcome: 'The run resolves triage_outside_prs to false (unset, layer default) and says so, lists #31 under the unlabeled bucket, and prints no row for a pull request: no line or table cell starts with a [PR] prefix and #32 is not listed as intake. It runs no pull-request listing command.' | ||
| execution: | ||
| prompt: "/work-items:triage\n\nInteractive session. The adapter's \"List items\" read for this repository returns two open items, both with no labels, no blockers and blocked_by_wont_do_count 0: issue #31 'CSV export drops the last row', opened 2026-10-02 by a reporter; and pull request #32 'Add a cache layer to the export', opened 2026-10-03 by an outside contributor (author_association NONE). Use this listing; there is no other tracker to read. Print the attention-view table only; build no board page." | ||
| allowed_tools: [Bash, Read, Glob, Grep] | ||
| max_turns: 15 | ||
| timeout_seconds: 240 | ||
| graders: | ||
| - name: lists-issue | ||
| type: regex | ||
| pattern: '#31\b' | ||
| target: last_message | ||
| arm: both | ||
| - name: reports-setting-false | ||
| type: regex | ||
| pattern: '\btriage_outside_prs\b[^\n]*?\bfalse\b' | ||
| flags: 'i' | ||
| target: last_message | ||
| arm: both | ||
| - name: no-pr-rows | ||
| type: regex | ||
| pattern: '^[ \t>]*(?:\|[^\n]*(?:\[PR\]|#32\b)|(?:[-*+]|\d+[.)])[ \t]+[^\n]*(?:\[PR\]|#32\b)|`?\[PR\])' | ||
| flags: 'm' | ||
| match: not_contains | ||
| target: last_message | ||
| arm: both | ||
| - name: no-pr-listing-call | ||
| type: tool_used | ||
| tool: Bash | ||
| input_match: '\bgh[ \t]+pr[ \t]+list\b|\bgh[ \t]+search[ \t]+prs\b|\bgh[ \t]+api\b[^\n]*?(?:/pulls\b|\bpullRequests\b)' | ||
| min: 0 | ||
| max: 0 | ||
| arm: both |
22 changes: 22 additions & 0 deletions
22
plugins/work-items/evals/tr-outside-prs-unset-lists-no-pr/samples/lists-issue.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": "Settings: triage_outside_prs=false (default)\n\n| Bucket | Item |\n|---|---|\n| unlabeled | #31 CSV export drops the last row |", | ||
| "why": "issue row" | ||
| }, | ||
| { | ||
| "answer": "Unlabeled: #31 CSV export drops the last row (opened 2026-10-02).", | ||
| "why": "prose line" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": "Nothing needs triage.", | ||
| "why": "drops the issue" | ||
| }, | ||
| { | ||
| "answer": "Unlabeled: #310 Rework the importer.", | ||
| "why": "another number" | ||
| } | ||
| ] | ||
| } |
65 changes: 65 additions & 0 deletions
65
plugins/work-items/evals/tr-outside-prs-unset-lists-no-pr/samples/no-pr-listing-call.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "bash \"/p/skills/triage/scripts/parse-concern-value.sh\" docs/conventions/work-items.yaml triage_outside_prs" | ||
| } | ||
| } | ||
| ], | ||
| "why": "settings read" | ||
| }, | ||
| { | ||
| "answer": [], | ||
| "why": "uses the given listing" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "gh pr list --state open --json number,title,author" | ||
| } | ||
| } | ||
| ], | ||
| "why": "lists PRs" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "gh api repos/o/r/pulls?state=open" | ||
| } | ||
| } | ||
| ], | ||
| "why": "lists PRs through the API" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "gh api \"repos/o/r/pulls?state=open\"" | ||
| } | ||
| } | ||
| ], | ||
| "why": "quoted API path" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "gh search prs --repo o/r --state open" | ||
| } | ||
| } | ||
| ], | ||
| "why": "PR search" | ||
| } | ||
| ] | ||
| } |
38 changes: 38 additions & 0 deletions
38
plugins/work-items/evals/tr-outside-prs-unset-lists-no-pr/samples/no-pr-rows.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": "Settings: triage_outside_prs=false (default)\n\n| Bucket | Item |\n|---|---|\n| unlabeled | #31 CSV export drops the last row |", | ||
| "why": "issue only" | ||
| }, | ||
| { | ||
| "answer": "| unlabeled | #31 CSV export drops the last row |\n\nOutside-PR intake is off, so no [PR] rows: #32 is not listed.", | ||
| "why": "mentions the prefix in prose, lists no PR row" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": "| Bucket | Item |\n|---|---|\n| unlabeled | [issue] #31 CSV export drops the last row |\n| unlabeled | [PR] #32 Add a cache layer to the export |", | ||
| "why": "PR table row" | ||
| }, | ||
| { | ||
| "answer": "Unlabeled:\n- [issue] #31 CSV export drops the last row\n- [PR] #32 Add a cache layer to the export", | ||
| "why": "PR list line" | ||
| }, | ||
| { | ||
| "answer": "[PR] #32 Add a cache layer to the export", | ||
| "why": "bare PR line" | ||
| }, | ||
| { | ||
| "answer": "| Item | Type | Title |\n|---|---|---|\n| #31 | issue | CSV export drops the last row |\n| #32 | PR | Add a cache layer to the export |", | ||
| "why": "PR row through a type column" | ||
| }, | ||
| { | ||
| "answer": "1. [issue] #31 CSV export drops the last row\n2. [PR] #32 Add a cache layer to the export", | ||
| "why": "numbered list" | ||
| }, | ||
| { | ||
| "answer": "- `[PR]` #32 Add a cache layer to the export", | ||
| "why": "prefix in backticks" | ||
| } | ||
| ] | ||
| } |
26 changes: 26 additions & 0 deletions
26
plugins/work-items/evals/tr-outside-prs-unset-lists-no-pr/samples/reports-setting-false.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": "Settings: triage_outside_prs=false (default)", | ||
| "why": "settings line" | ||
| }, | ||
| { | ||
| "answer": "`triage_outside_prs` is unset, so it resolves to `false` and the view lists no pull request.", | ||
| "why": "prose" | ||
| }, | ||
| { | ||
| "answer": "triage_outside_prs: false (layer default)", | ||
| "why": "colon form" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": "Settings: triage_outside_prs=true (repository)", | ||
| "why": "wrong value" | ||
| }, | ||
| { | ||
| "answer": "Outside PRs are not listed.", | ||
| "why": "names no setting" | ||
| } | ||
| ] | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.