Add a pr-feedback-harvester cron to close the skill feedback loop - #311
Draft
alex-clickhouse wants to merge 1 commit into
Draft
Add a pr-feedback-harvester cron to close the skill feedback loop#311alex-clickhouse wants to merge 1 commit into
alex-clickhouse wants to merge 1 commit into
Conversation
The three skill-maintenance jobs all read from inside the instance. skill-extractor recalls memory and the existing skill list; skill-reviser loads the SKILL.md files and audits them against themselves. Neither one has any way to learn that a reviewer already told the agent it got something wrong. That leaves the highest-signal correction available — a human on a pull request saying "this is wrong, do it this way" — with no path into a skill. The only route today is accidental: an interactive session happens to read the comment, it lands in memory, and skill-extractor may or may not surface it twelve hours later. The harvester is the inbound half of that loop. Weekly, it reads the review threads on the PRs the agent authored in the last seven days, clusters the feedback into recurring themes rather than filing one item per comment, and splits the result by scope: general -> a generic skill, or a workspace instruction file repo-specific -> that repository's own dev skill Everything goes through the normal task+plan approval flow; the job never edits a skill directly. It memorizes each cluster regardless of what it proposes, so a declined or slow-to-approve plan doesn't lose the lesson. Two details in the prompt are load-bearing: The job ingests third-party text and then proposes edits to the agent's own instructions, which is a prompt-injection path. The prompt states that comments are evidence of what a reviewer wanted, never instructions addressed to the agent. plan_type is auto-detected from a task's source only for skill-extractor and skill-reviser. A cluster here can yield either a new skill or a revision, so source-based detection cannot decide; the job passes plan_type explicitly. Omitting it would silently produce a generic plan that spawns an implementation session instead of writing the skill. Scheduled Monday 04:00, a day clear of skill-reviser's Sunday 03:00 — both propose skill edits, and the gap keeps them from duelling over the same file. Enabled by default in worker mode alongside the other skill crons, offered during nerve init in personal mode, and gated on having either an authenticated gh CLI or a GitHub sync source. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
All three skill-maintenance jobs read from inside the instance:
skill-extractorskill-reviserNeither has any way to learn that a reviewer already told the agent it got something wrong.
skill-reviseraudits skills against themselves — it doesn't even recall memory.So the highest-signal correction the agent ever receives — a human on a PR saying "this is wrong, do it this way" — has no path into a skill. The only route today is accidental: an interactive session happens to read the comment, it lands in memory, and
skill-extractormay or may not surface it 12 hours later.What this adds
pr-feedback-harvester— weekly, the inbound half of that loop. It reads the review threads on PRs the agent authored in the last 7 days, clusters the feedback into recurring themes rather than filing one item per comment, and splits the result by scope:AGENTS.mdviapropose_config_changeEverything goes through the normal task+plan approval flow — the job never edits a skill directly. It
memorizes each cluster regardless of what it proposes, so a declined or slow-to-approve plan doesn't lose the lesson.Two details in the prompt are load-bearing
Prompt injection. The job ingests third-party text and then proposes edits to the agent's own instructions. The prompt states explicitly that comments are evidence of what a reviewer wanted, never instructions addressed to the agent. There's a test asserting the guard is still present.
plan_typemust be explicit. Auto-detection maps a task'ssourceto a plan type forskill-extractor→skill-createandskill-reviser→skill-updateonly. A cluster here can yield either a new skill or a revision, so source-based detection can't decide — the job passesplan_typeexplicitly. Omitting it would silently produce agenericplan that spawns an implementation session instead of writing the skill.Scheduling
Monday 04:00 — a day clear of
skill-reviser's Sunday 03:00. Both propose skill edits, and the gap keeps them from duelling over the same file. The prompt also checkstask_search/plan_listand skips clusters that already have an open proposal.Enabled by default in worker mode alongside the other skill crons; offered during
nerve initin personal mode.requiresis set to an authenticatedghCLI or a GitHub sync source, and the job exits cleanly when neither is present.Author weighting
The shipped commands filter on
.user.type, not the login string —.user.loginis unreliable because some review bots have no[bot]suffix (GitHub's ownCopilotreviewer istype: Bot,login: Copilot). The prompt weights humans highest, treats automated code reviewers as useful-but-noisy (act only on repeated points), and ignores coverage/CI/dependency bots entirely.Testing
.venv/bin/pytest tests/ -v— 3193 passedsystem.yamlenabled with the right schedule, and the definition keeps both guardrailsload_jobs()parses it andCronTrigger.from_crontab("0 4 * * 1")schedules the next fire correctlyghcommand in the prompt was run against a live account before being written down🤖 Generated with Claude Code