Repository navigation
feat(work-items): route an idea issue to an interview before triage explores it #11893
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
melodic-automation-lanes
wants to merge
2
commits into
main
from
work-resolve-issue/issue-11869-41281674150-1
Closed
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
7 changes: 7 additions & 0 deletions
7
.changes/work-items/work-resolve-issue-issue-11869-5d2c9a17.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| bump: minor | ||
| --- | ||
|
|
||
| ### Added | ||
|
|
||
| - triage: an idea issue, a feature request that does not say what it is for, now goes to `/planning:interview` and is parked with `needs-spec` before any code search, ledger check, verification or research (#11869). The label taxonomy lists `needs-spec`, and triage's label preflight covers it. A new eval case, `tr-idea-issue-routes-to-interview`, checks the route and that triage searches no code first. |
47 changes: 47 additions & 0 deletions
47
plugins/work-items/evals/tr-idea-issue-routes-to-interview/case.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| schema_version: "1.1" | ||
| name: tr-idea-issue-routes-to-interview | ||
| description: 'Idea issue: /work-items:triage named on item #1, a raw feature request ("We should add a plugin system to the ledger.") that states no problem, user or outcome, routes the item to /planning:interview and names needs-spec as the label that parks it. It runs no exploration first: no code search, no source read, no research. The prompt authorizes mutations up front, so the run is not held at the direction gate. The case grades the route and the absence of exploration only, not the park itself: the local-markdown tracker has no label listing, so the needs-spec preflight in context/apply-outcome.md may stop the label edit, and no grader reads the final labels of item #1. A run needs --scaffold and --allow-tools Bash,Write,Edit.' | ||
| tags: [triage, idea-issue, must-not] | ||
| plugins: ["../.."] | ||
| expected_outcome: 'The run reads item #1, judges it an idea issue because it says nothing about what the plugin system is for, names /planning:interview as the next step and names needs-spec as the label that parks it. It searches no code and reads nothing under src/ before or after routing. Whether the park lands is not graded: on this tracker, which has no label listing, the needs-spec preflight may stop the label edit.' | ||
| context: | ||
| scaffold_script: scaffold.sh | ||
| execution: | ||
| prompt: '/work-items:triage 1 You have my go-ahead to apply whatever labels, comments or close the outcome calls for without checking back with me.' | ||
| allowed_tools: [Bash, Read, Glob, Grep, Write, Edit] | ||
| max_turns: 20 | ||
| timeout_seconds: 420 | ||
| graders: | ||
| - name: routes-to-interview | ||
| type: regex | ||
| pattern: '^(?=[\s\S]*/planning:interview)(?=[\s\S]*\bneeds-spec\b)' | ||
| target: last_message | ||
| arm: both | ||
| - name: no-grep | ||
| type: tool_used | ||
| tool: Grep | ||
| input_match: '^(?![\s\S]*/work-items/)' | ||
| min: 0 | ||
| max: 0 | ||
| arm: both | ||
| - name: no-glob | ||
| type: tool_used | ||
| tool: Glob | ||
| input_match: '^(?![\s\S]*(?:\.work|/work-items/))' | ||
| min: 0 | ||
| max: 0 | ||
| arm: both | ||
| - name: no-source-read | ||
| type: tool_used | ||
| tool: Read | ||
| input_match: '"file_path":"(?:[^"\\]|\\.)*/src/' | ||
| min: 0 | ||
| max: 0 | ||
| arm: both | ||
| - name: no-bash-search | ||
| type: tool_used | ||
| tool: Bash | ||
| input_match: '(?:"command":"|[ \t;&(]|\\[nt])(?:(?:grep|rg|ag)[ \t]|find[ \t]+[./~]|git[ \t]+(?:grep|log)\b)|(?:[ \t/"]|\\[nt])src(?![\w-])' | ||
| min: 0 | ||
| max: 0 | ||
| arm: both | ||
98 changes: 98 additions & 0 deletions
98
plugins/work-items/evals/tr-idea-issue-routes-to-interview/samples/no-bash-search.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "/p/tools/work-item-tracker/work-item-tracker.sh get-item 'local-markdown:local/markdown#1'" | ||
| } | ||
| } | ||
| ], | ||
| "why": "reads the item through the seam" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "/p/tools/work-item-tracker/work-item-tracker.sh comment 'local-markdown:local/markdown#1' --body 'Run /planning:interview to find out what this is for.'" | ||
| } | ||
| } | ||
| ], | ||
| "why": "prose 'find out' in a comment" | ||
| }, | ||
| { | ||
| "answer": [], | ||
| "why": "nothing" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "grep -rn plugin ." | ||
| } | ||
| } | ||
| ], | ||
| "why": "greps the checkout" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "git log --oneline -20" | ||
| } | ||
| } | ||
| ], | ||
| "why": "reads history" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "find . -name '*.js'" | ||
| } | ||
| } | ||
| ], | ||
| "why": "finds files" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "cat src/export.js" | ||
| } | ||
| } | ||
| ], | ||
| "why": "reads the source" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "cd /w\ngrep -rn plugin ." | ||
| } | ||
| } | ||
| ], | ||
| "why": "greps on a second line" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Bash", | ||
| "input": { | ||
| "command": "ls src" | ||
| } | ||
| } | ||
| ], | ||
| "why": "lists the source" | ||
| } | ||
| ] | ||
| } |
54 changes: 54 additions & 0 deletions
54
plugins/work-items/evals/tr-idea-issue-routes-to-interview/samples/no-glob.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Glob", | ||
| "input": { | ||
| "pattern": ".work-items/*.md" | ||
| } | ||
| } | ||
| ], | ||
| "why": "lists the item store" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Glob", | ||
| "input": { | ||
| "pattern": ".work/lanes/*.json" | ||
| } | ||
| } | ||
| ], | ||
| "why": "looks for the lane config the scope rule reads" | ||
| }, | ||
| { | ||
| "answer": [], | ||
| "why": "nothing" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Glob", | ||
| "input": { | ||
| "pattern": "src/**/*.js" | ||
| } | ||
| } | ||
| ], | ||
| "why": "lists the source" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Glob", | ||
| "input": { | ||
| "pattern": "**/*plugin*" | ||
| } | ||
| } | ||
| ], | ||
| "why": "explores the checkout" | ||
| } | ||
| ] | ||
| } |
57 changes: 57 additions & 0 deletions
57
plugins/work-items/evals/tr-idea-issue-routes-to-interview/samples/no-grep.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Grep", | ||
| "input": { | ||
| "pattern": "needs-spec", | ||
| "path": "/cache/plugins/work-items/reference" | ||
| } | ||
| } | ||
| ], | ||
| "why": "searches the plugin's own references" | ||
| }, | ||
| { | ||
| "answer": [], | ||
| "why": "nothing" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Grep", | ||
| "input": { | ||
| "pattern": "plugin" | ||
| } | ||
| } | ||
| ], | ||
| "why": "searches the checkout" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Grep", | ||
| "input": { | ||
| "pattern": "export", | ||
| "path": "/w/src" | ||
| } | ||
| } | ||
| ], | ||
| "why": "searches the source" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Grep", | ||
| "input": { | ||
| "pattern": "plugin", | ||
| "path": "/w/.work-items" | ||
| } | ||
| } | ||
| ], | ||
| "why": "cluster search over the item store" | ||
| } | ||
| ] | ||
| } |
32 changes: 32 additions & 0 deletions
32
plugins/work-items/evals/tr-idea-issue-routes-to-interview/samples/no-source-read.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Read", | ||
| "input": { | ||
| "file_path": "/w/.work-items/1.md" | ||
| } | ||
| } | ||
| ], | ||
| "why": "reads the item" | ||
| }, | ||
| { | ||
| "answer": [], | ||
| "why": "nothing" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Read", | ||
| "input": { | ||
| "file_path": "/w/src/export.js" | ||
| } | ||
| } | ||
| ], | ||
| "why": "reads the source" | ||
| } | ||
| ] | ||
| } |
26 changes: 26 additions & 0 deletions
26
plugins/work-items/evals/tr-idea-issue-routes-to-interview/samples/routes-to-interview.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": "#1 \"Add a plugin system\" is an idea issue: it says nothing about what the plugin system is for. Posted a comment naming `/planning:interview` as the next step, applied `needs-spec` and removed `needs-triage`.", | ||
| "why": "names the interview and the park label" | ||
| }, | ||
| { | ||
| "answer": "Parked #1 with needs-spec.\n\nNext: /planning:interview, to settle what the plugin system is for before anyone explores it.", | ||
| "why": "both on separate lines" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": "Triaged #1: enhancement, priority: medium, briefed and routed to needs-human.", | ||
| "why": "state-machined, no interview" | ||
| }, | ||
| { | ||
| "answer": "Searched src/ for an existing extension point and found none. Next: /planning:interview.", | ||
| "why": "no needs-spec park" | ||
| }, | ||
| { | ||
| "answer": "Applied needs-spec to #1.", | ||
| "why": "no interview route" | ||
| } | ||
| ] | ||
| } |
39 changes: 39 additions & 0 deletions
39
plugins/work-items/evals/tr-idea-issue-routes-to-interview/scaffold.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| #!/usr/bin/env bash | ||
| # A checkout bound to the offline local-markdown tracker holding one open idea issue: a feature | ||
| # request carrying the raw needs-triage marker that names something to build but says nothing | ||
| # about what it is for. The checkout also holds source code an exploring triage would search. | ||
| set -euo pipefail | ||
|
|
||
| git init -q -b main | ||
| git config user.name eval | ||
| git config user.email eval@example.invalid | ||
|
|
||
| printf '.work/\n' >.gitignore | ||
| printf '# ledger\n\nLedger service.\n' >README.md | ||
| mkdir -p .work-items src | ||
| cat >src/export.js <<'JS' | ||
| export function exportCsv(rows) { | ||
| return rows.map((row) => row.join(",")).join("\n"); | ||
| } | ||
| JS | ||
| cat >.work-item-tracker.json <<'JSON' | ||
| {"schema_version": "1.0", "provider": "local-markdown", "config": {"lease_ttl_hours": 24, "storage_dir": ".work-items"}} | ||
| JSON | ||
| cat >.work-items/1.md <<'MD' | ||
| --- | ||
| id: "local-markdown:local/markdown#1" | ||
| number: 1 | ||
| title: "Add a plugin system" | ||
| state: "open" | ||
| assignees: [] | ||
| labels: ["needs-triage"] | ||
| type: null | ||
| parent: null | ||
| url: "file://.work-items/1.md" | ||
| created_at: "2026-10-01T08:00:00Z" | ||
| --- | ||
|
|
||
| We should add a plugin system to the ledger. | ||
| MD | ||
| git add .gitignore README.md src .work-item-tracker.json .work-items | ||
| git commit -q -m "chore: scaffold ledger" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.