docs(AGENTS): mandate pre-merge verification#378
Conversation
Codify the pre-merge checks so future agents do not require the maintainer to ask "did we run it / did we test it / does CI gate this?". Three checks, all mandatory, all done before declaring a PR ready: 1. Run it locally (with `bun install` to sync deps when verifying). 2. Test it (existing test or new smoke test in the same PR). 3. Confirm CI parallel runs green via `gh run list --branch <branch>`. Surfaced after PR #364 (coverage harness) was pushed without local run, missed a Biome formatting failure on first push and a missing- dep failure on local invocation — both 60-second checks the agent should have done before declaring the PR ready. Adversarial review: verified the new section reads as one of three self-contained blocks, doesn't duplicate the adjacent "Always Improve" or "Review Your Own Change Adversarially" sections, and states the discipline in actionable terms (numbered, with explicit ✓/✗ output convention). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Warning Rate limit exceeded
To keep reviews running without waiting, you can enable usage-based add-on for your organization. This allows additional reviews beyond the hourly cap. Account admins can enable it under billing. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Adds a new Pre-Merge Verification policy to AGENTS.md so contributors and agents must close the local-run/test/CI loop before marking runnable changes as ready. This fits the repo’s broader “dark factory” workflow by tightening process around scripts, workflows, CLIs, and build targets.
Changes:
- Adds a
## Pre-Merge Verificationsection toAGENTS.md. - Requires three checks before merge readiness: run the artifact, verify test coverage, and confirm CI is green.
- Introduces a standard verification-status string for final summaries / PR comments.
|
|
||
| A PR that adds or modifies a runnable artifact (script, workflow, CLI entry point, build target) is not mergeable until **all three** of these are done — without waiting to be asked: | ||
|
|
||
| 1. **Run it.** Invoke the new entry point locally and confirm exit 0 plus the expected artifact (file, output, exit code). For coverage harnesses this means actually generating a report; for workflow changes, push to a branch and watch the run. `bun install` to sync deps when verifying a PR is part of the work — the "ask before bun install" rule applies to *unrelated* installs. |
|
|
||
| 1. **Run it.** Invoke the new entry point locally and confirm exit 0 plus the expected artifact (file, output, exit code). For coverage harnesses this means actually generating a report; for workflow changes, push to a branch and watch the run. `bun install` to sync deps when verifying a PR is part of the work — the "ask before bun install" rule applies to *unrelated* installs. | ||
| 2. **Test it.** Either an existing test exercises the new path, or a new spec / smoke test is added in the same PR. "I trust the author / I trust the tooling" doesn't count. | ||
| 3. **CI parallel.** After push, run `gh run list --branch <branch>` and confirm every triggered workflow is GREEN. If lint / tests / publish-check fail, fix and re-push *before* declaring ready for merge. |
| 2. **Test it.** Either an existing test exercises the new path, or a new spec / smoke test is added in the same PR. "I trust the author / I trust the tooling" doesn't count. | ||
| 3. **CI parallel.** After push, run `gh run list --branch <branch>` and confirm every triggered workflow is GREEN. If lint / tests / publish-check fail, fix and re-push *before* declaring ready for merge. | ||
|
|
||
| State the verification result explicitly in the final summary or PR comment: "ran locally: ✓; CI green: ✓; smoke test added: ✓". If anything is ✗, fix it. Don't declare done with known failures. |
| 2. **Test it.** Either an existing test exercises the new path, or a new spec / smoke test is added in the same PR. "I trust the author / I trust the tooling" doesn't count. | ||
| 3. **CI parallel.** After push, run `gh run list --branch <branch>` and confirm every triggered workflow is GREEN. If lint / tests / publish-check fail, fix and re-push *before* declaring ready for merge. | ||
|
|
||
| State the verification result explicitly in the final summary or PR comment: "ran locally: ✓; CI green: ✓; smoke test added: ✓". If anything is ✗, fix it. Don't declare done with known failures. |
|
@brianmhunt I would like to share at this point some options to save tokens:
|
Summary
Adds a new `## Pre-Merge Verification` section to AGENTS.md mandating three checks before declaring any PR with a runnable artifact ready:
Why
Surfaced today on PR #364 (coverage harness): pushed without local run; first push lint-failed on a Biome formatting violation, then local invocation failed on a missing dep. Both 60-second pre-checks the agent should have done before declaring the PR ready. User had to prompt with "did we run it?" to surface gaps that should have been mandatory upfront.
The dark-factory thesis (small team + AI agents) only works if agents close the verification loop without prompting. Long PR threads where a maintainer has to chase basic checks defeat the throughput model.
What changed
One new section in AGENTS.md, between "Always Improve" and "Review Your Own Change Adversarially". 10 lines of prose, explicit output convention ("ran locally: ✓; CI green: ✓; smoke test added: ✓").
Test plan