Add LLM-assisted reviewer router (Kubernetes-only v1) by ChrisJBurns · Pull Request #5637 · stacklok/toolhive

ChrisJBurns · 2026-06-25T14:00:17Z

Summary

Today's .github/CODEOWNERS assigns reviewers purely by path glob, which makes every listed owner a required, blocking, notified reviewer. That conflates two different jobs — gatekeeping ("must approve") and awareness ("want to know") — and the result is notification fatigue: owners get pinged on churn within paths they own but no longer actively work, while static lists quietly go stale.

This PR introduces an advisory, context-aware reviewer router that rides on top of the existing CODEOWNERS gate (which is left untouched and still governs merges). Instead of a coarse glob, an LLM reads a natural-language ownership doc and requests reviewers based on what actually changed — e.g. it can require the controller owner on reconcile-logic changes but skip the full operator list for a pure CRD-doc regeneration.

Scope is intentionally limited to Kubernetes changes for this first iteration so the mechanism can prove out before expanding.

.github/REVIEWERS.md — natural-language ownership doc (Kubernetes-only for v1) with per-area conditions that let routing be more precise than a glob (skip doc-regen/chart-version-bump/generated-only changes; always include @ChrisJBurns on controller changes).
.github/workflows/assign-reviewers.yml — a pull_request-triggered workflow that runs claude-code-action (same pattern as the existing issue-triage workflow) to request the appropriate reviewers.

Type of change

New feature

Test plan

Manual testing (describe below)

This is CI/automation config (no Go code). Verification so far: YAML structure and action/SHA pins match the existing issue-triage.yml and claude.yml workflows. End-to-end behavior (the action requesting reviewers on a real Kubernetes PR) needs to be observed once merged or tested on a throwaway PR, since the workflow only runs with repository secrets available.

Does this introduce a user-facing change?

No. This only affects the reviewer-assignment experience for contributors; it does not change ToolHive's runtime behavior.

Special notes for reviewers

Security model. The workflow uses pull_request (not pull_request_target, which this repo deliberately avoids), so fork/dependabot PRs receive no ANTHROPIC_API_KEY and the Claude step skips — those fall back to the deterministic CODEOWNERS gate. The token is scoped to pull-requests: write and allowed_tools is restricted to reading repo files plus the GitHub API. Routing is advisory: the worst a prompt-injected diff can do is request the wrong reviewer, which CODEOWNERS still gates against.

Known follow-ups (intentionally out of scope here):

allowed_tools is currently mcp__github__*; it should be narrowed to just the reviewer-request tool. Left broad for the v1 trial.
Routing covers Kubernetes only; other areas (CLI, vMCP, security, AI gateway, …) fall through to the existing CODEOWNERS default and will be added once this proves out.
A future "slim gate" CODEOWNERS only becomes worthwhile once router coverage is broad enough to make the full file redundant.

Generated with Claude Code

Coarse path-glob CODEOWNERS forces every listed owner onto a PR as a required, blocking, notified reviewer, conflating gatekeeping with awareness. Owners get pinged on irrelevant churn within paths they own, while static lists go stale. Introduce an advisory, context-aware reviewer router that complements a slim deterministic gate: - REVIEWERS.md: natural-language ownership doc with Required/Notify tiers and per-area conditions (e.g. skip doc-regen, scope to a sub-feature) so routing can be more precise than a glob. - assign-reviewers.yml: pull_request-triggered workflow using a brain/hands split. The "brain" reasons over the diff with no token and no write access; a deterministic "hands" script performs all API calls. - assign-reviewers.sh: validates the brain's picks against an allowlist of handles in REVIEWERS.md, drops the author, requests reviewers, and upserts one notification comment. - CODEOWNERS.proposed: slim deterministic gate for sensitive paths only. Fork PRs receive no secret and fall back to the deterministic gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Pare the first iteration down to a single area so the mechanism can prove out before expanding. REVIEWERS.md now routes only Kubernetes changes; everything else falls through to the default owner. Since routing is advisory and the merge gate stays in CODEOWNERS, the brain/hands split is more machinery than this scope needs: let the action request reviewers directly with tools restricted to reading files and the GitHub API, and a token scoped to pull-requests:write. Remove the deterministic assignment script and the precompute step. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The k8s-only advisory router leaves the existing CODEOWNERS as the deterministic gate, so a slimmed-gate draft is unused until the router's coverage grows. Remove it; revisit when expanding scope. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

codecov · 2026-06-25T14:07:03Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.31%. Comparing base (baa8bb4) to head (b98c92f).
⚠️ Report is 20 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5637      +/-   ##
==========================================
+ Coverage   69.96%   70.31%   +0.35%     
==========================================
  Files         651      649       -2     
  Lines       66505    66170     -335     
==========================================
  Hits        46529    46529              
+ Misses      16624    16288     -336     
- Partials     3352     3353       +1

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Address three review findings: - M4: read REVIEWERS.md from the base commit, not the PR head. The head can rewrite the routing rules and the handle allowlist in the same PR, so the "edits are gated" guarantee only held at merge, not at CI evaluation. Mirror how GitHub evaluates CODEOWNERS (from the base branch). - H1: narrow allowed_tools from mcp__github__* to PR inspection plus reviewer request, and correct the security comment so it no longer overstates what the step can do. - L1: add an author_association guard for consistency with claude.yml. The exact GitHub MCP reviewer-request tool name (H2) is still unverified and needs a live run to confirm reviewers are actually requested. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The GitHub MCP server has no reliable human-reviewer-request tool, so the prior MCP-based write was unverified and likely a no-op. Switch to a deterministic gh api call: - Brain (Claude) now reads only local files (trusted REVIEWERS.md + a precomputed changed-file list) and writes a decision file. Its tools are Read/Glob/Grep/Write only -- no Bash, no GitHub access. - Hands: a final bash step validates the decision against the REVIEWERS.md allowlist, drops the author, and POSTs to .../requested_reviewers. This makes the write mechanism verifiable on first run (resolves H2) while keeping the brain unable to act on the PR directly. The script is inline in the workflow (retest.yaml style), not a separate file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The base-ref overwrite guarded against an author rewriting the routing rules in their own PR. But routing is advisory: rerouting only changes who is requested, and an author cannot approve their own PR or grant an out-of-org approval, so CODEOWNERS still gates the merge. The incentive runs toward getting the correct reviewer. Drop the fetch/git-show/overwrite and simply read the head copy; skip routing when REVIEWERS.md is absent. If CODEOWNERS is ever slimmed so the router carries gate weight, base-ref reads must come back. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Reviewer requests alone give no visibility into why someone was pinged, which matters for a new LLM-driven mechanism and for spotting REVIEWERS.md mis-tuning. Have Claude attach a one-sentence reason per pick, and have the hands step upsert a single marker comment listing the reviewers and reasons. The comment is found by an HTML-comment marker and edited in place on later pushes rather than re-posted, so it does not spam the PR. Reasons are filtered through jq (and stripped of '@') so free-text cannot inject mentions or break the comment payload. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The first live run failed two ways: - Without github_token the action falls back to OIDC and dies ("Could not fetch an OIDC token"). Pass github_token like issue-triage.yml does. - `allowed_tools` is not a valid input on this action version and was silently ignored. Move the restriction into claude_args --allowedTools. The brain stays handless at the tool level: allowed tools are Read/Glob/Grep/Write only, so Claude cannot act on the PR despite the token being present; the gh api step still performs the request. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Throwaway commit to exercise the happy path of the reviewer router on a Kubernetes change. Will be reverted.

The exit-5 crash was in the validation filter, not Claude's output (which was a valid flat object array). In `$allow | index(.handle | ascii_downcase)` the inner `.handle` is evaluated against $allow (the array), so it tried to index an array with "handle" and aborted. Bind the lowercased handle to a variable and index $allow with that instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The blanket "Required: <everyone>" lists were copied from CODEOWNERS, so the router reproduced the exact spam it exists to prevent (a one-line operator change pinged all 8 k8s owners). Remove the blanket lists entirely: a reviewer is requested ONLY when a specific rule names them for that kind of change, and unmatched changes request no one. CODEOWNERS remains the safety net. Update the prompt to match (no default list; minimal, rule-named set). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@ChrisJBurns

Controller/reconcile-logic changes now route to @ChrisJBurns and @JAORMX. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Drop the prep step that fetched the changed-file list via the API. Give the brain a read-only, scoped `Bash(git diff:*)` tool and full checkout history (fetch-depth 0) so it determines what changed itself, and can inspect actual diffs for content-based rules. Tools stay limited to file reads + git diff (no other commands, no GitHub access), so the brain still cannot act on the PR; the deterministic gh api step continues to do the request. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

A two-dot `git diff base HEAD` against the checked-out merge ref pulled in files that changed on main since the branch point (a stray CRD-types file showed up in routing). Switch to the three-dot form between explicit base and head SHAs, which diffs from the merge-base and shows only what the PR itself changed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-06-26T18:45:12Z

🤖 Reviewer router requested the following based on .github/REVIEWERS.md:

@JAORMX — Controller reconcile-logic change in cmd/thv-operator/controllers/mcpgroup_controller.go

Advisory only — merge gating is governed by CODEOWNERS.

ChrisJBurns and others added 3 commits June 24, 2026 18:29

github-actions Bot added the size/S Small PR: 100-299 lines changed label Jun 25, 2026

github-actions Bot added size/S Small PR: 100-299 lines changed and removed size/S Small PR: 100-299 lines changed labels Jun 25, 2026