docs(skills): implement v1 agent-authored-content quality program - #679
Conversation
Co-Authored-By: Warp <agent@warp.dev>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
This PR was generated with Warp. Comment    |
Co-Authored-By: Warp <agent@warp.dev>
Co-Authored-By: Warp <agent@warp.dev>
warp-agent-staging
Bot
changed the title
|
@warp-agent-staging[bot] I'm starting a first review of this pull request. You can view the conversation on Warp. I completed the review and no human review was requested for this pull request. Comment Powered by Oz |
![warp-factories[bot]](https://avatars.githubusercontent.com/in/2278701?s=60&v=4){kind=small}
There was a problem hiding this comment.
This PR implements the GROW-6092 v1 agent-doc quality program (policy module, CI gates, independent review workflow, metrics, and skill wiring). The core library and unit tests are solid and green, but several contract/enforcement gaps would break the program once required checks go live, and the PR's own risk classification is inconsistent with the allowlist it introduces.
Found: 2 critical, 2 important, 2 suggestions, 0 nits
Request changes
![warp-for-oss[bot]](https://avatars.githubusercontent.com/in/3450139?s=60&v=4){kind=small}
There was a problem hiding this comment.
Overview
This PR adds a v1 quality program for agent-authored docs changes, including policy helpers, CI gates, workflow dispatch, review guidance, and measurement scripts. The intent is coherent, but several implementation details break or bypass the proposed gates.
Concerns
- The PR declares
Risk: low, but this diff changes CI workflows, Python policy scripts, commands, tests, and agent behavior. That is outside the strict low-risk allowlist and should beengineering-review-required. - The new CI contract check will fail ordinary pull requests and can deadlock engineering-review-required agent PRs before their human gate state exists.
- The docs override path trusts PR-authored metadata instead of a trusted reviewer/approval source, so an author can forge or weaken the human gate.
- The outcome metrics can report success without proving the post-rollout gate requirements actually passed.
Security
- The override authorization path reads both the claimed override reviewer and the reviewer allowlist from PR-controlled inputs, which lets untrusted PR content satisfy a human-review gate.
- The review workflow installs a mutable external Debian package with
sudoin the same job that receives the Warp API key.
Verdict
Found: 4 critical, 13 important, 0 suggestions
Request changes
Comment /warp-agent-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
There was a problem hiding this comment.
This PR implements the GROW-6092 v1 agent-doc quality program (shared policy module, required CI gates, independent review workflow, metrics, and skill wiring). The pure library/unit tests are solid and green, but the CI wiring of the PR contract and the agent-review gate have enforcement gaps that would break docs PRs once the checks are required, and this PR's own declared risk is inconsistent with the allowlist it introduces.
Found: 2 critical, 2 important, 1 suggestions, 0 nits
Request changes
- Gate the documentation-risk contract check on the warpy-factory label and re-run on PR body edits/labeling, so non-agent PRs are never failed. - Load the authorized-override allowlist from a trusted base ref instead of the PR's own checkout, and derive review/check-pass signals from live GitHub data (gh api) instead of trusting caller-supplied flags. Fail closed on an empty allowlist and on a missing current head SHA. - Verify a real, current, non-blocking GitHub review exists at the exact head SHA before Agent docs review can pass. - Refresh and commit a verified valid_paths.json snapshot with real source_repository/source_sha provenance, and add --require-provenance to fail the required gate closed on incomplete provenance. - Require all in-scope current-window PRs to have passed required checks before evaluate_outcome() can report a v1 outcome pass. - Replace the hand-maintained PR-producing skill manifest with mechanical discovery of every direct PR-creation code path, and wire the contract into every real path found (validate_ui_refs/style_lint auto-fix PRs, release_updates, weekly-404-monitor). - Capture and persist the real pre-rollout baseline (2026-08-01 to 2026-08-30, 62 PRs) from the existing human_review_feedback.jsonl log and live gh PR data. Co-Authored-By: Warp <agent@warp.dev>
There was a problem hiding this comment.
This PR implements the GROW-6092 v1 agent-doc quality program (shared policy module, required CI gates, independent review workflow, metrics, baseline, and skill wiring). The rework fixed several earlier holes (warpy-factory gating of the contract check, trusted-ref allowlist loading, live review signals, verify_review_signal, real snapshot provenance, evaluate_outcome gate-pass requirement, discovery-based manifest, and a real pre-rollout baseline), and the unit suites are green. Remaining issues still break or weaken the program once checks are required: the human engineering gate is enforced too early in CI, docs overrides can be forged from PR body text alone, this PR's own risk is still misclassified as low, and a few provenance/compression/metrics edges remain open.
Found: 2 critical, 5 important, 3 suggestions, 0 nits
Request changes
![warp-agent-staging[bot]](https://avatars.githubusercontent.com/in/2196476?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review verdict: rejected (self-review constraint)
Final re-review found 4 critical and 5 important blockers. The final independent review job failed for this head, and the repository’s protected-branch configuration still does not require either new deterministic check.
The remaining defects violate the committed GROW-6092 specification: the human technical gate is enforced during ordinary CI and cannot refresh after an engineer approval; docs overrides are forgeable from PR-body text; the independent-review gate does not validate its required parseable signal; and snapshot, compression, and metrics edges produce incorrect outcomes.
Validation: npm run typecheck, npm run build, internal link checking, whitespace checking, and the relevant Python suites passed. The failures below are behavioral gaps not covered by those tests.
[SIGNAL:pr-review]
PR: 679
Branch: factory/ai-content-quality-v1
Head SHA: fe6bbf8
Skill: factory-review
Verdict: Request changes
Critical: 4
Important: 5
Categories: human-gate, authorization, review-signal, branch-protection, risk-routing, provenance, compression, metrics
GitHub prevents this authenticated PR author from submitting REQUEST_CHANGES on its own pull request. This structured review records the required rejected verdict; the independent reviewer has already submitted CHANGES_REQUESTED on the same head.
There was a problem hiding this comment.
This PR implements the GROW-6092 v1 agent-doc quality program (shared policy module, required CI gates, independent review workflow, metrics/baseline, and skill wiring). The rework improved several earlier holes (label-gated contract check, trusted-ref allowlist loading, live review signals, verify_review_signal, real snapshot provenance, discovery-based manifest, evaluate_outcome gate-pass requirement, and a real pre-rollout baseline), and the unit suites are green.
Re-validated the declared Risk: low against the low-risk allowlist and the actual diff: this is still a risk misclassification. Independently re-checked the human gate on this head and confirmed it still fails both legitimate paths (source-owner approval after GitHub clears reviewRequests, and a forgeable PR-body override).
Found: 2 critical, 5 important, 2 suggestions, 0 nits
Request changes
[SIGNAL:pr-review] {"date":"2026-09-03","pr":"679","branch":"factory/ai-content-quality-v1","head_sha":"0d10dcfa1b03f43576ac5be66f3dec9d6f1d5de5","skill_used":"review-docs-pr","verdict":"Request changes","critical":2,"important":5,"suggestions":2,"nits":0,"top_categories":["human-gate (2)","authorization (1)","review-signal (2)"]}
Co-Authored-By: Warp <agent@warp.dev>
Co-Authored-By: Warp <agent@warp.dev>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
This PR implements the v1 agent-authored-content quality program (deterministic scorecard, shared PR contract, independent review pass, and feedback/measurement loop) entirely as internal docs-repo tooling, skill prose, and CI workflow changes. No files under src/content/docs/ are touched, so it makes no public product claim. I independently re-validated the declared Risk: low against the low-risk allowlist in .agents/references/doc-quality-policy.md: the diff adds no new/changed feature page, adds no VERIFY marker, and is confined to internal Docs-team tooling/CI/skill maintenance with no product-behavior claim, so low is the correct classification (not a misclassification). I ran the full doc_quality_policy (87 tests), validate_ui_refs --changed/provenance (11 tests), and improve-drafting-skills metrics (27 tests) suites locally — all 125 pass. I also recomputed the committed pre-rollout baseline report from the committed JSONL via compute_metrics.py and it reproduces byte-for-byte, confirming that claim. All four touched/added GitHub Actions workflow YAML files parse correctly, and all changed/added Python files compile cleanly.
One minor accuracy nit: the PR body's Verification section claims "108 unit tests, all passing", but the actual current count across the three cited suites (doc_quality_policy + validate_ui_refs + improve-drafting-skills) is 125 — likely stale from before the rework commits added test_manifest.py/test_verify_review_signal.py and other cases. Not blocking since all tests do pass; just worth correcting the number before merge.
Found: 0 critical, 0 important, 0 suggestions, 1 nit.
Verdict: Approve with nits.
[SIGNAL:pr-review] {"branch": "factory/ai-content-quality-v1", "critical": 0, "date": "2026-09-04", "head_sha": "0cb15b3b9671af4f12b6705904910caeecc6d2d3", "important": 0, "nits": 1, "pr": "679", "reviewer_login": "github-actions[bot]", "skill_used": "n/a", "suggestions": 0, "top_categories": ["verification_accuracy (1)"], "verdict": "Approve with nits"}
Co-Authored-By: Warp <agent@warp.dev>
There was a problem hiding this comment.
This is the bootstrapping PR for the v1 agent-doc-quality program itself (the doc_quality_policy module, the review-docs-pr contract, new required CI jobs, and the independent-review/engineering-review workflows). It touches no src/content/docs/ page and makes no product claim, so the declared Risk: low is correctly classified under the low-risk allowlist's "internal Docs-team tooling, skill, or CI workflow maintenance" carve-out — this is not a misclassification. The ## Documentation risk and ## Unverified claims sections parse cleanly (validated directly against policy.validate_pr_contract), there are no unaccounted {/* VERIFY: ... */} markers (every VERIFY: occurrence in the diff is documentation/regex/test-fixture text describing the marker syntax itself, not a live unresolved claim), and I reproduced all 108 referenced unit tests plus the style-lint, broken-link, and validate_ui_refs --self-test verification steps locally — all pass, confirming the PR's Verification section is accurate. valid_paths.json's source_sha/source_repository/generated_at provenance fields are populated (not null).
One important structural defect: .github/workflows/docs-engineering-approval.yml is added as a completely empty (0-byte) file — git show confirms its blob is the empty-tree hash. It is not referenced by name anywhere else in this diff, in ci.yml, in request-engineering-review.yml, or in agent-docs-review.yml, and the PR body's own "Verification" section (which claims "All three GitHub Actions workflows ... parse as valid YAML") doesn't even mention it or the other new request-engineering-review.yml. It looks like scaffolding left behind from renaming to request-engineering-review.yml during the rework pass. An empty file under .github/workflows/ has no on:/jobs: and will surface as an invalid-workflow warning in the Actions UI on every future push; it should be removed (or filled in and wired up) before merge.
Found: 0 critical, 1 important, 1 suggestion, 0 nits. Recommendation: Request changes (blocking on the orphaned empty workflow file per the v1 contract's blocking rule; risk classification, VERIFY accounting, and every mechanically-checkable claim otherwise check out).
[SIGNAL:pr-review] {"branch": "factory/ai-content-quality-v1", "critical": 0, "date": "2026-09-04", "head_sha": "6b586fbde7879f5fe7ed3c5ad55463e069c399c4", "important": 1, "nits": 0, "pr": "679", "reviewer_login": "github-actions[bot]", "skill_used": "doc_quality_policy", "suggestions": 1, "top_categories": ["orphaned/empty workflow file (1)", "PR description completeness (1)"], "verdict": "Request changes"}
Co-Authored-By: Warp <agent@warp.dev>
There was a problem hiding this comment.
PR #679 implements the v1 agent-authored-content quality program: the warpy-factory marker/PR-body contract, deterministic CI scorecard, independent review-docs-pr gate, and a feedback/measurement loop. The diff touches only .agents/ skill tooling, .github/workflows/, and internal specs/logs -- zero files under src/content/docs change, so no public product claim is made. I re-validated the declared Risk: low against the low-risk allowlist in .agents/references/doc-quality-policy.md: the PR adds no feature page, changes no command/API/UI/default/permission/availability/billing/security/self-hosting/integration behavior, and is exactly 'internal Docs-team tooling, skill, or CI workflow maintenance that makes no public product claim' -- the declared risk is correct, not a misclassification. I ran the structural contract check the way CI actually invokes it (check_pr_contract.py --body <body>, relying on its origin/main...HEAD -- src/content/docs autodiscovery): it passes cleanly with 0 VERIFY markers and 0 changed docs files, matching the PR's own verification claims. I also independently ran all cited test suites (test_policy.py, test_check_pr_contract.py, test_check_compression_contract.py, test_verify_review_signal.py, test_manifest.py, validate_ui_refs/test_changed_mode.py, improve-drafting-skills metrics/baseline tests), the broken-link checker (--internal-only), validate_ui_refs.py --changed --require-provenance against the committed snapshot, and YAML-parsed all four changed/added workflow files -- everything passes as claimed. One suggestion below on a latent false-positive edge case in the VERIFY-marker regex, and a minor discrepancy in the reported test count; neither blocks this PR.
Found: 0 critical, 0 important, 1 suggestion, 1 nit.
Recommendation: Approve with nits.
[SIGNAL:pr-review] {"branch": "factory/ai-content-quality-v1", "critical": 0, "date": "2026-09-04", "head_sha": "cbc1f9244b2c15c5a2d363c9ba08a80b81fd22c7", "important": 0, "nits": 1, "pr": "679", "reviewer_login": "github-actions[bot]", "skill_used": "doc_quality_policy", "suggestions": 1, "top_categories": ["VERIFY-marker regex false-positive scope (1)", "test-count precision in PR body (1)"], "verdict": "Approve with nits"}
Co-Authored-By: Warp <agent@warp.dev>
There was a problem hiding this comment.
Review summary
PR #679 bootstraps the v1 agent-authored-content quality program (doc_quality_policy module, review-docs-pr/CI contract, request-engineering-review and refresh-ui-paths workflows) entirely as internal docs-repo tooling and skill prose; no files under src/content/docs/ are touched, so the declared "low" risk is correctly classified against the low-risk allowlist and is not a misclassification. All new/changed Python unit test suites (policy.py, check_pr_contract.py, check_compression_contract.py, verify_review_signal.py, test_manifest.py, validate_ui_refs --changed/--self-test, improve-drafting-skills metrics) pass locally, all four changed GitHub Actions workflow files parse as valid YAML, and running the actual required checks (style_lint.py --changed, validate_ui_refs.py --changed --require-provenance, check_pr_contract.py) against this exact head produces zero findings. Found: 0 critical, 0 important, 1 suggestion, 0 nits.
Findings
.agents/skills/doc_quality_policy/finalize_pr_contract.py:86— 💡 [SUGGESTION]--engineering-review-statusis constrained tonot-applicable/pending/approvedhere, but the corresponding PR-body field is parsed as an unvalidated free-form string inpolicy.parse_documentation_risk_section, andvalidate_pr_contractnever checks its value. A manually-edited PR body withEngineering review status: boguswould currently passcheck_pr_contract.pysilently. Consider validating this field (or its absence when risk islow) invalidate_pr_contractthe same wayriskanddocs_overrideare validated.
Verdict
Approve with nits
Review signal
[SIGNAL:pr-review] {"branch": "factory/ai-content-quality-v1", "critical": 0, "date": "2026-09-04", "head_sha": "99d0481618f55ae8289be76a99a447c1b6daff2d", "important": 0, "nits": 0, "pr": "679", "reviewer_login": "github-actions[bot]", "skill_used": "doc_quality_policy", "suggestions": 1, "top_categories": ["schema-validation-gap (1)"], "verdict": "Approve with nits"}
| build.add_argument("--rationale", required=True) | ||
| build.add_argument("--source-files", help="comma-separated file@sha list") | ||
| build.add_argument("--reviewers", help="comma-separated GitHub handles") | ||
| build.add_argument("--engineering-review-status", choices=("not-applicable", "pending", "approved")) |
There was a problem hiding this comment.
💡 [SUGGESTION] --engineering-review-status is constrained to not-applicable/pending/approved here, but the corresponding PR-body field is parsed as an unvalidated free-form string in policy.parse_documentation_risk_section, and validate_pr_contract never checks its value. A manually-edited PR body with Engineering review status: bogus would currently pass check_pr_contract.py silently. Consider validating this field (or its absence when risk is low) in validate_pr_contract the same way risk and docs_override are validated.
Co-Authored-By: Warp <agent@warp.dev>
There was a problem hiding this comment.
Review summary
PR #679 bootstraps the v1 agent-authored-content quality program (doc_quality_policy module, review-docs-pr/CI contract, request-engineering-review and refresh-ui-paths workflows) entirely as internal docs-repo tooling, skill prose, and CI workflows; no files under src/content/docs/ are touched, so the declared "low" risk is correctly classified against the low-risk allowlist and is not a misclassification. All 18 changed/new Python modules compile and their unit test suites pass (130 tests), all 4 changed/new GitHub Actions workflow files parse as valid YAML, and the actual required checks for this exact head (style_lint.py --changed, validate_ui_refs.py --changed --require-provenance, the doc_quality_policy test suite, and check_pr_contract.py) all pass, matching the green CI status on this PR. The final commit (a020e72) also fixed a real bug from the prior review cycle: the GitHub review-state mapping previously sent "Approve with nits" as a COMMENT event, which would not supersede an earlier CHANGES_REQUESTED review from the same reviewer, and it now correctly maps to APPROVE; it also added the missing engineering_review_status validation flagged as a suggestion in the prior review pass. Found: 0 critical, 0 important, 0 suggestions, 0 nits.
Findings
- No inline findings.
Verdict
Approve
Review signal
[SIGNAL:pr-review] {"branch": "factory/ai-content-quality-v1", "critical": 0, "date": "2026-09-04", "head_sha": "a020e723918436bdd0944b31f33104950bae146f", "important": 0, "nits": 0, "pr": "679", "reviewer_login": "github-actions[bot]", "skill_used": "doc_quality_policy", "suggestions": 0, "top_categories": [], "verdict": "Approve"}
Co-Authored-By: Warp <agent@warp.dev>
Co-Authored-By: Warp <agent@warp.dev>
There was a problem hiding this comment.
Review summary
The independent agent completed its review for this commit.
Findings
- unverifiable-operational-claim (1)
Verdict
Approve
Review signal
[SIGNAL:pr-review] {"branch": "factory/ai-content-quality-v1", "critical": 0, "date": "2026-09-04", "head_sha": "4f7caf8f3162f622d084afd35b56075bfe5001b2", "important": 0, "nits": 0, "pr": "679", "reviewer_login": "github-actions[bot]", "skill_used": "create_pr", "suggestions": 1, "top_categories": ["unverifiable-operational-claim (1)"], "verdict": "Approve"}
This branch was successfully deployed
1 participant
Summary
Implements the v1 agent-authored documentation quality program for
warpdotdev/docs.Docs editorial qualityrunsstyle_lint.py --changed;Docs technical referencesvalidates changed UI references, snapshot provenance, andVERIFYaccounting.warpy-factory, documentation-risk metadata, and unverified-claim accounting. Internal Docs tooling with no public product claim follows the low-risk Docs review path; technical documentation requests source-owner review without requiring a GitHub approval to merge.Agent docs reviewruns on the PR head, requires a matching review and structured signal from the reviewing agent, and rejects stale, blocking, or conflicting review output.Documentation risk
Risk: low
Rationale: This PR changes internal Docs-team tooling, skills, CI workflows, and measurement scripts. It adds no public product claim.
Docs override: none
Unverified claims
None — this PR adds no unresolved public product claims.
Verification
This change has no user-facing UI surface.
Docs editorial quality,Docs technical references,Agent docs review, build/link checks, and CodeQL for this PR.mainruleset requiresDocs editorial quality,Docs technical references, andAgent docs review, in addition toverification/cla-signed.npm run typecheck,npm run build, and internal link checking run in the CI build job.Originating thread: https://warpdev.slack.com/archives/C09BVK0PL3Y/p1788384154207199
Co-Authored-By: Warp agent@warp.dev