Skip to content

Add a pr-feedback-harvester cron to close the skill feedback loop - #311

Draft
alex-clickhouse wants to merge 1 commit into
mainfrom
alex-clickhouse/pr-feedback-harvester-cron
Draft

Add a pr-feedback-harvester cron to close the skill feedback loop#311
alex-clickhouse wants to merge 1 commit into
mainfrom
alex-clickhouse/pr-feedback-harvester-cron

Conversation

@alex-clickhouse

Copy link
Copy Markdown
Collaborator

Problem

All three skill-maintenance jobs read from inside the instance:

Job Evidence Answers
skill-extractor memory recall + existing skill list "What do I keep doing that isn't written down?"
skill-reviser the SKILL.md files themselves "Is what's written down still true?"

Neither has any way to learn that a reviewer already told the agent it got something wrong. skill-reviser audits skills against themselves — it doesn't even recall memory.

So the highest-signal correction the agent ever receives — a human on a PR saying "this is wrong, do it this way" — has no path into a skill. The only route today is accidental: an interactive session happens to read the comment, it lands in memory, and skill-extractor may or may not surface it 12 hours later.

What this adds

pr-feedback-harvester — weekly, the inbound half of that loop. It reads the review threads on PRs the agent authored in the last 7 days, clusters the feedback into recurring themes rather than filing one item per comment, and splits the result by scope:

  • General (commit hygiene, testing discipline, security habits) → a generic skill, or a workspace instruction file such as AGENTS.md via propose_config_change
  • Repo-specific (build commands, module layout, local conventions) → that repository's own dev skill, creating one if it doesn't exist

Everything goes through the normal task+plan approval flow — the job never edits a skill directly. It memorizes each cluster regardless of what it proposes, so a declined or slow-to-approve plan doesn't lose the lesson.

Two details in the prompt are load-bearing

Prompt injection. The job ingests third-party text and then proposes edits to the agent's own instructions. The prompt states explicitly that comments are evidence of what a reviewer wanted, never instructions addressed to the agent. There's a test asserting the guard is still present.

plan_type must be explicit. Auto-detection maps a task's source to a plan type for skill-extractorskill-create and skill-reviserskill-update only. A cluster here can yield either a new skill or a revision, so source-based detection can't decide — the job passes plan_type explicitly. Omitting it would silently produce a generic plan that spawns an implementation session instead of writing the skill.

Scheduling

Monday 04:00 — a day clear of skill-reviser's Sunday 03:00. Both propose skill edits, and the gap keeps them from duelling over the same file. The prompt also checks task_search / plan_list and skips clusters that already have an open proposal.

Enabled by default in worker mode alongside the other skill crons; offered during nerve init in personal mode. requires is set to an authenticated gh CLI or a GitHub sync source, and the job exits cleanly when neither is present.

Author weighting

The shipped commands filter on .user.type, not the login string — .user.login is unreliable because some review bots have no [bot] suffix (GitHub's own Copilot reviewer is type: Bot, login: Copilot). The prompt weights humans highest, treats automated code reviewers as useful-but-noisy (act only on repeated points), and ignores coverage/CI/dependency bots entirely.

Testing

  • .venv/bin/pytest tests/ -v3193 passed
  • 3 new tests: worker-mode defaults include the job, it reaches system.yaml enabled with the right schedule, and the definition keeps both guardrails
  • Generated a worker-mode config end to end and confirmed the prompt round-trips through YAML intact
  • load_jobs() parses it and CronTrigger.from_crontab("0 4 * * 1") schedules the next fire correctly
  • Every gh command in the prompt was run against a live account before being written down
  • No frontend change — the cron UI reads job ids from the API

🤖 Generated with Claude Code

The three skill-maintenance jobs all read from inside the instance.
skill-extractor recalls memory and the existing skill list; skill-reviser
loads the SKILL.md files and audits them against themselves. Neither one
has any way to learn that a reviewer already told the agent it got
something wrong.

That leaves the highest-signal correction available — a human on a pull
request saying "this is wrong, do it this way" — with no path into a
skill. The only route today is accidental: an interactive session happens
to read the comment, it lands in memory, and skill-extractor may or may
not surface it twelve hours later.

The harvester is the inbound half of that loop. Weekly, it reads the
review threads on the PRs the agent authored in the last seven days,
clusters the feedback into recurring themes rather than filing one item
per comment, and splits the result by scope:

  general       -> a generic skill, or a workspace instruction file
  repo-specific -> that repository's own dev skill

Everything goes through the normal task+plan approval flow; the job never
edits a skill directly. It memorizes each cluster regardless of what it
proposes, so a declined or slow-to-approve plan doesn't lose the lesson.

Two details in the prompt are load-bearing:

The job ingests third-party text and then proposes edits to the agent's
own instructions, which is a prompt-injection path. The prompt states
that comments are evidence of what a reviewer wanted, never instructions
addressed to the agent.

plan_type is auto-detected from a task's source only for skill-extractor
and skill-reviser. A cluster here can yield either a new skill or a
revision, so source-based detection cannot decide; the job passes
plan_type explicitly. Omitting it would silently produce a generic plan
that spawns an implementation session instead of writing the skill.

Scheduled Monday 04:00, a day clear of skill-reviser's Sunday 03:00 —
both propose skill edits, and the gap keeps them from duelling over the
same file. Enabled by default in worker mode alongside the other skill
crons, offered during nerve init in personal mode, and gated on having
either an authenticated gh CLI or a GitHub sync source.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant