Conversation
verify.sh runs a deterministic, non-interactive baseline for a Dependabot PR (tidy, codegen drift, UI dist drift, binary and dev-server smoke, actions pinning/coverage, docker base image, and more), compares failures against the base branch to separate pre-existing issues, runs agent-generated per-PR checks on both versions, and writes result.json and comment.md. post-comment.sh is the only script that writes to GitHub and only upserts one comment. Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
…otes Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
- run/go_tool keep command echoes and tool downloads out of captured output - pass gh's config dir through the hermetic env so action.yml comparison works - require the goreleaser-cross snapshot for cgo-tagged updates (required_for_tags) - replace the CGO-off cross-compile (cannot build internal/dev_server/db/backup) with an opt-in goreleaser-cross snapshot - dev-server smoke passes a placeholder access token instead of relying on env - report go test cache hits Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
…ample Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
…m and a scratch prefix go-npm falls back to $npm_config_prefix/bin because newer npm has no 'npm bin', so a plain local npm ci tries to write into the global prefix. Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
…semantics Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
…verifier-9b50 Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
Verdicts are now block, needs-human, incomplete, or safe-to-merge. Checks report fail, decide (one exact question with evidence), incomplete, or info. A person is asked only for a real decision. Tool and environment gaps are reported as incomplete. A gate must pass: a pre-existing failure never satisfies it. High tier alone no longer asks a person. New checks collect upstream release notes and diffs, compare transitive dependencies and licenses, verify go directive changes, and compare action outputs, input defaults, runtime, and workflow permissions. Each failing check can carry a machine-applicable fix recipe in result.json. release-snapshot pulls the private goreleaser-cross image first and reports incomplete if the pull fails. docker-image builds the static CGO musl binary that the release ships and runs the dev server in the image. Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
…wer standard The skill and the playbooks now say what a thorough reviewer must make sure of for each ecosystem and risk tier, what the baseline checks cover, what the agent must do, and when to ask a person. An impact review can state that no local check can prove a reachable change, which turns it into a decision. Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
…the same change Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
…e-snapshot If the private ghcr.io/launchdarkly/goreleaser-cross image cannot be pulled, release-snapshot uses the public goreleaser/goreleaser-cross v1.24.2 image, pinned by digest, with sha256-pinned musl.cc toolchains mounted at /musl. If neither image can be pulled, the check is incomplete. The summary names the image. The details list each built target and the fidelity gaps: the public image and musl.cc replace the LaunchDarkly image, the binaries are not executed, and goreleaser build does not run the Docker and Homebrew steps. Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
The pr-829-release-targets-build fixture expects safe-to-merge and a release-snapshot pass for all 8 targets. pr-829-gate-not-satisfied now makes both images unpullable, so it still proves that a missing image gives incomplete. replay.sh can match text in a check summary. Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
- verdict.jq: if each PR finding of a list check (actionlint, golangci-lint, prettier, actions-pinning, npm ls) also occurs on base, the failure is pre-existing. Before, a PR that removed one actionlint finding got a block (#717, #721). - golangci-lint: run with --allow-parallel-runners. A non-zero exit with no finding is incomplete, not a failure with 0 findings. - ui-npm-ci: take the conflict from npm error lines only. An npm warn line named the wrong peer on #729. - deps.py: a package that already had an install script on base adds no new script (esbuild on #779). - ui-dep-usage: join package names with "and". - pr-state: for a conflicting PR, do not say that the checks ran on the PR merged into main. - verify.sh: save the baseline profile. --phase generated and --phase render use it instead of the default profile. Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
- classify.sh: when workflows pin an action at several old refs, record each ref that the PR replaces. The old version is the lowest of them. Before, #718 compared release-secrets v1.2.0 with itself, and #717 compared only setup-go v5 with v6. - actions_analysis.py: compare each replaced ref with the new ref. An output or input counts as removed only if an old action.yml declared it. release-please-action declares no outputs and sets them at run time, which gave a false block on #722. - actions-coverage.sh: read fields with the unit separator. Tab is IFS whitespace, so empty fields collapsed and shifted the runtime into the wrong field (#717, #721). Show a missing default as "none". Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
- pr-disclosure: for a grouped update, or a PR with more than one direct update, list the direct updates that the PR title and body do not name, and ask one decision. #779 names dompurify and uuid, but also moves @launchpad-ui/components, launchdarkly-js-client-sdk, react-router, and vite. - upstream.py: use the from_ref and to_ref from classify. Map golang.org/x/* to the github.com/golang mirror, and use the commit messages as notes when a Go module has no release notes. For npm monorepo packages (react-router, @launchpad-ui/*), read the package CHANGELOG.md and use <name>@<version> tags. - verify.sh --pr-meta FILE: read the PR title and body from a file, for replays that need them without GitHub. Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
New fixtures: #717 safe-to-merge (setup-go from v4 and v5), #718 compares release-secrets v1.0.1 with v1.2.0, #721 needs-human with the python-version question, #722 needs-human (not block), and #779 block with the undisclosed updates and no esbuild question. The #831 fixture expects the new wording. The impact-review copies drop notes about the old verifier bugs. replay.sh supports pr_meta, decisions_exclude, and updates expectations. run.sh adds 16 unit tests with stub tools: the findings subset rule, golangci-lint lock contention, npm warn lines, install scripts on base, pr-disclosure, pr-state wording, classify with mixed refs, and actions-coverage with a stub gh. Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
…e sources Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
The disclosure evidence says "none of them" instead of "0 of them". The ERESOLVE summary drops the node_modules path that npm prints on the line after the conflict. Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
Try <name>@<version>, v<version>, and <version>, and use the first tag that exists. vite tags v8.2.1 and launchdarkly-js-client-sdk tags 3.9.5, so #779 got no compare link or notes for them. Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
…ced updates - pr-disclosure strips bot blocks (<!-- NAME --> … <!-- /NAME -->, such as the Cursor Bugbot summary) and the <details> sections that quote upstream notes. On #737, golang.org/x/term was named only in the Bugbot summary, and the check passed. - A direct Go update that a named update requires at the new version or higher is forced (deps.py forced walks go mod graph). It is a note, not a decision. On #621, mock 0.6.0 needs x/term 0.34.0 through x/tools and x/net. - The check calls an update grouped only if the title says so. - deps.py licenses also reports the package name, version, and dev flag. Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
npm audit flags each package that depends on a vulnerable package. On #723 it flagged @launchpad-ui/core only through navigation and react-router, whose advisories base has too, and the check asked about a new advisory that does not exist. The check now compares GHSA IDs. Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
- A CHANGELOG section ends at the first release heading at or below the old version, not after 400 lines. If no such heading exists, the coverage is partial, with the lowest version that the notes reach. - If a version has no tag (@launchpad-ui/core 0.59.17), read the CHANGELOG on the default branch and skip the compare link. - Scan up to 10 pages of releases, and stop at the old version. - The summary gives full, partial, and missing coverage instead of "notes for N of N". On a low-tier update, it does not ask for an impact review. - The breaking keyword also matches removed packages, components, exports, props, inputs, and outputs. Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
…old image tags - actions_analysis: evaluate defaults as github.com does. A GHES fallback (github.server_url == 'https://github.com' && X || Y) is X, so the setup-python token default is not a change (#721). The question names the workflows that use the action. - license-changes: one item per package family, which says dev-only build tools: "Accept MPL-2.0 for lightningcss 1.33.0 and 11 lightningcss-* package(s) (dev-only build tools)?" (#779). - verdict.jq: a guard regression or a failing gate already shows that a change reaches ldcli, so the missing discriminating proof is not an incomplete item (#723, #729, #779). - release-snapshot: accept a tagged release image. A conflicting PR is tested at its old head, and trees from before #629 use goreleaser-cross:v1.24.2 instead of a digest (#621). Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
govulncheck -format json reports each advisory as reachable, in an imported package, or in a required module. Only a new reachable advisory fails, as before. The summary now lists the fixed advisories by level and the reachable advisories that stay (#737 fixes nine and leaves GO-2026-5970 and GO-2026-5320). Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
New fixtures: #621 (forced x/term update, not grouped, release snapshot on the old tree) and #737 (x/term named only in the Bugbot summary). The #723 fixture expects no audit question and full note coverage. The #779 fixture expects full note coverage, GHSA IDs, and the short license question. The #721 fixture expects no token default in the question. run.sh adds 12 unit tests with stub tools: bot text and <details> in pr-disclosure, forced Go updates, npm audit advisory IDs, note coverage (partial CHANGELOG, missing tag, release scan past 3 pages), the license family question, impact proof with a guard regression, and the govulncheck levels. Each one fails on the code before this change. Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
… govulncheck levels Co-authored-by: Ramon Niebla <nieblara@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requirements
scripts/dependency-pr/test/run.sh: 71 unit tests.scripts/dependency-pr/test/replay.sh: 14 replays of past PRs.)Related issues
Dependabot PRs stay open for a long time, and a green CI run does not mean that a PR is safe. #720 merged green, but its generated code was never regenerated, and the regenerated code does not compile.
Describe the solution you've provided
This PR adds a toolkit that verifies a Dependabot PR to the standard of a diligent reviewer. After a run, a person does not have to verify anything again. A person only answers the decisions that the comment asks for. The toolkit does not approve or merge.
scripts/dependency-pr/verify.sh --pr N(or--branch B) runs without prompts. It makes abaseworktree and aprworktree (the PR merged into the base tip), classifies the updates, and runs the baseline checks. It writesresult.json,comment.md, and logs.safe-to-merge(exit 0),needs-human(exit 1, one yes-or-no question for each decision, with evidence),block(exit 1, with the fix), orincomplete(exit 2, a check or review did not run).npm ls), a PR is also pre-existing if each of its findings occurs on base. A tool or environment failure isincomplete, neverfail.go generatedrift with a build of the regenerated code, and the go directive.dist/drift, ERESOLVE, unused dependencies, and new advisories.pr-disclosure). Bot summaries and quoted release notes do not count as disclosure. A Go update that a named update requires is a note, not a decision.@launchpad-ui/*, vite), they come from the packageCHANGELOG.md. Forgolang.org/x/*, they come from the commit messages on the GitHub mirror. The summary says whether the notes cover the full range, part of it, or none of it.ghcr.io/launchdarkly/goreleaser-crossimage when GHCR credentials can pull it. Otherwise, it uses the publicgoreleaser/goreleaser-cross:v1.24.2image, pinned by digest, with sha256-pinned musl.cc toolchains at/musl. If neither image can be pulled, the check is incomplete. The check details state the fidelity gaps: the public image and musl.cc replace the LaunchDarkly image, the binaries are not executed, and the Docker and Homebrew steps do not run.agent/impact.json) and generated checks. A discriminating check counts only if it fails on the old version and passes on the new version.commandplus expectedpaths) inresult.json.fixes[]. A laterapply-fixes.shcan use the recipes.post-comment.shis the only script that writes to GitHub. The caller runs it. It edits one comment and does not post if the PR head moved..agents/skills/verify-dependency-pr/, with playbooks for gomod, the UI, the npm wrapper, GitHub Actions, and Docker. AGENTS.md points to it and corrects the pre-commit section.Validation (local, no comments posted)
The runs merged each PR into
main@9197f92, on a VM with Docker and no GHCR credentials. The 2026-10-06 rows come after the fixes for the bugs that two runs on all 16 open Dependabot PRs found.npm cifails: react-router 8.0.1 needs react >= 19.2.7.npm cifails, and new GHSA advisories appear. The description does not name 4 of the direct updates (react-router 8, vite 8, and two others). License question: "Accept MPL-2.0 for lightningcss 1.33.0 and 11 lightningcss-* package(s) (dev-only build tools)?"npm cifails on the react 19.2.6 peer. No new advisory ID, so no audit question.The replays cover each row above except #719, #729, and #830. Each fixture added for a bug fails on the code before the fix. They also cover #720 (block), #639 stale dist (block, with a rebuild recipe), a low-risk patch #635 (safe to merge), and the #829 false safe case with no pullable image (incomplete).
Describe alternatives you've considered
jq, and small Python helpers, so that a GitHub workflow can call the same files.internal/dev_server/db/backup, so the toolkit uses goreleaser-cross.zig ccfor the cross-compile without an image. It cannot link the darwin targets without a macOS SDK, so the toolkit uses the public goreleaser-cross image.Additional context
verify.shin apull_requestjob with a read-only token. Post from a separateworkflow_runjob. Do not usepull_request_targetwith a PR checkout.go-generate-driftcheck reports the chore(deps): bump github.com/oapi-codegen/oapi-codegen/v2 from 2.4.1 to 2.7.1 #720 drift onmainas pre-existing. This PR does not fix it.