Is there an existing issue or feature request for this?

• I have searched the existing issues and feature requests.

What area does this improvement affect?

New Feature

What actions does this improvement affect?

Pull Request, Commit

Description of the idea or improvement

Add a language- and framework-independent verification stage between Bugbot candidate detection and publication. The diff defines which changes are under review. The pinned repository revision and relevant project evidence establish whether a change introduces or exposes a reachable defect. The product outcome is a PR review in which every published or reopened defect has a traceable basis, and a reviewer can distinguish a refuted candidate, an unresolved question, and an interrupted run.

Scope and ownership

• Apply the rule to every eligible finding that would become active, including new findings, updated active findings, reopened findings, and findings counted as overflow. Cover ordinary and framed PR reviews, incremental review and recheck, and the supported Commit review path with or without an owning PR. Preserve the existing canonical PR ownership and fallback rules so push and PR jobs cannot both publish the same review. Apply explicit evidence requirements to any proposed fixed or obsolete transition on an existing finding. Preserve human dismissal authority.
• Extend the existing Bugbot analysis and publication capability. Generalize the existing candidate verification boundary and reuse the current context, source identity, partitioning, finding identity, history, freshness, approval and presentation contracts. A file extension may select a source adapter or an evidence method. Eligibility for verification follows the finding, regardless of language, framework, severity rank or whether a comment slot is available.

End-to-end flow

Pin the review target and source → detect candidate hypotheses in the diff
→ merge equivalent candidates and plan evidence → retrieve project evidence
→ independently try to refute each candidate → classify every eligible candidate
→ recheck freshness → publish and reconcile one result for that revision

1. Bind the review. Carry the existing repository, target, event, reviewed head and applicable base or comparison identity through detection, evidence reads, decisions and publication. Use base evidence when required to establish whether the change introduced the defect or made it reachable. A broad checkout, a branch name, or a similarly named CI check is insufficient provenance. For PR merge checkouts or CI builds using a synthetic merge, require verified source or tree equivalence before treating the result as evidence for the relevant revision and path.
2. Prepare a testable candidate. Preserve its stable identity and changed-code anchor. Capture the change, triggering input or state, reachable path and user-visible or security impact. Apply existing admission rules and deduplicate equivalent hypotheses before verification. Retain relevant evidence across duplicates and distinguish different root causes at the same location. Apply ranking and the visible comment limit after verification. Every retained overflow finding that affects review status must also be confirmed. A candidate rejected by existing scope or shape rules does not consume a verifier slot. Malformed or missing required batch results remain workflow failures under the existing integrity rules.
3. Acquire evidence according to the claim. The application owns bounded, read-only evidence acquisition and supplies the result to the verifier. The model may ask for evidence through a closed, typed request within the bound review target. Support immutable source reads and bounded searches across changed and unchanged files, callers, guards, configuration, tests and contracts. Use the actual import and resolved dependency or runtime version for framework, language and API claims. Use matching native CI job, suite or step evidence for claims about a build or test, and distinguish a matching execution from a test that was skipped, cached, run against another revision, or irrelevant. A green check in general carries no verdict about a specific claim. Where a new reproduction is needed and supported, run only a preauthorized, trusted, bounded verification command or probe with pinned inputs in an isolated environment. Existing source and CI evidence should work on repositories without an execution profile. Unavailable or denied evidence yields a bounded diagnostic and an honest disposition.
4. Try to falsify the claim. Use a fresh verification task, independent of the detection conversation, with the concrete hypothesis, its source binding and evidence actually fetched by the application. Examine earlier guards, alternate paths, matching executed counterevidence, effective configuration and contract exceptions. Keep a defect when the evidence establishes the causal path and its impact. Verify the proposed replacement separately before offering an executable suggestion. Keep a confirmed defect with a prose explanation if the proposed replacement cannot be verified. Allow a short bounded additional evidence round for a material gap or conflict. The SDK verifies provenance and response integrity. The verifier evaluates whether the evidence supports or contradicts the claim. A valid link or a confidence score alone cannot establish the verdict.
5. Validate an explicit disposition for every eligible ID. Use confirmed, refuted or inconclusive. Confirmed requires evidence of the reachable defect and a valid anchor. Refuted requires evidence that contradicts a material premise, such as an effective guard, the correct versioned API behavior, or a matching execution that directly falsifies the predicted failure. Inconclusive is a plausible, eligible hypothesis whose decisive evidence is missing, unavailable, conflicting or outside the allowed evidence budget. It carries a reason and a practical next step. The application checks that IDs, scope, artifact identities and source revisions match the request, that referenced evidence was actually supplied, and that each required disposition occurs once. Treat forged, missing, duplicate or cross-run verdicts as an integrity failure. A verifier cannot introduce a new finding or change the historical identity to make its claim fit.
6. Publish once after all required work is accounted for. With an integral and current result, publish or reopen only confirmed findings. Discard refuted new candidates and summarize inconclusive ones without opening defect threads for them. If two defects are confirmed and one candidate is inconclusive, publish the two confirmed findings, update the canonical summary once and mark the review incomplete for automatic approval. Track diff coverage, candidate verification and provider publication separately.
   Automatic clean approval requires complete required coverage and verification, no actionable or unresolved finding state, and all existing approval gates. Keep existing branch protection and repository ownership rules. A workflow-level evidence integrity failure, missing required partition, supersession or cancellation before publication produces no new findings or resolution mutations. Keep the last verified result identified by its own revision and surface the failure or supersession through the existing run/check behavior. A publication failure after a provider write follows existing recovery and re-read semantics, records observed effects and never claims a clean result from intended writes.
7. Reconcile history and retries. Verify a proposed reopen, fixed or obsolete transition against the same pinned project evidence and the explicit retained historical ID. A contradicted historical premise may qualify as obsolete only through the established transition policy with explicit evidence. An unresolved question or omission preserves the existing thread state and any verification requirement. Honor verified human decisions and native thread facts. Recheck the same head after recoverable evidence becomes available and update existing comments without duplicate writes. A new commit supersedes in-flight work. Preserve a single publication owner and the existing race fences around irreversible effects.

Evidence, security and operating limits

• Use evidence records issued by the application and bound to the review and candidate. Include source kind and safe locator, such as repository plus commit, path and line range, a resolved dependency version and contract source, or a CI producer/job and reviewed tree. Record enough provenance to detect evidence swapping and support operator review. The published view uses concise safe links and a causal explanation. Telemetry may use counts, safe reason categories, timing and correlation identifiers. Keep raw prompts, command output, credentials and unnecessary repository content out of public and durable records.
• Resolve only authorized repositories, Git objects, paths, versioned dependency sources and CI producers. Treat repository content, logs and model output as untrusted data. Enforce traversal, symlink, link and credential boundaries and validate evidence before any provider write. Execution is opt-in through an existing or explicitly reviewed trusted repository capability with a disposable, isolated runtime and bounded output, time and access. Model text and PR content cannot authorize commands, new network destinations or credentials.
• Put per-candidate and aggregate limits on source reads, retained evidence, requests, tokens, time, concurrency and retry. Reuse existing frame and traffic accounting for ordinary and framed reviews and verify limit accounting at the actual adapter boundary. Deduplicate safe identical reads within the same pinned scope. Distinguish a candidate-level evidence limit from an invalid or interrupted review. Report the exhausted resource and next supported action. Set numerical limits and any rollout setting from pilot evidence in the owning SDD. Snapshot effective settings per review. Evidence provenance and approval integrity are required in every enabled mode.

What GitHub should show

• Pending. “Changes analyzed. Checking 3 candidate findings against this commit.” Link to the current run where available.
• Complete. “1 confirmed defect. 2 candidates refuted.” Include the confirmed comment with a source link and a short explanation of the reachable impact.
• Incomplete. “2 defects confirmed and published. 1 candidate still needs the test result for this commit.” Link to the affected job or give the supported next action. Show why automatic approval is unavailable.
• Failed or superseded. Identify the affected revision, the failure or newer revision, and the last verified review where available. Report whether any GitHub publication effects were actually observed.

Reuse the canonical status card, Copilot / Review, approval inputs and Job Summary. Keep candidate dispositions separate from durable historical finding states. Update status once per result using the existing locale and sanitization contracts. Keep operator diagnostics expandable and avoid publishing a separate comment for each rejected or inconclusive hypothesis.

Current limitations or challenges

The audit used Copilot v3.10.13 as the published source baseline (ace49e4). Re-establish the baseline at the selected development revision before implementation. Bugbot currently plans review work from the diff, passes surrounding project guidance and lets its reviewer inspect the checkout. Partition attestations and coverage establish which inputs the run processed. They do not establish which supporting files, dependency contracts or executions the reviewer consulted for a particular claim.

The existing generated-source verification is enabled mainly for partitions containing Python. It reuses the detection context and the configured reviewer model. Its response checks enforce identity, a complete decision set and nonempty evidence text for retained candidates. The application does not currently bind that evidence text to sources it retrieved for the candidate. In #914, the native job reported complete context, retained two candidates in that verification step and published both. The installed pytest fixture discovery excluded the defaulted arguments on which both findings depended. In #919, the exact reviewed source checks retained SQLite history and stops before the claimed failure. #898 documents the same wrong Unicode premise in ha-ui and Domus, with executable counterexamples.

Relevant implementation and contracts: candidate routing, existing verifier, verification response policy, publication workflow, and quality tooling. The audited historical reviews used GPT-6 Luna. The causal contribution of model choice remains an empirical question.

Expected impact

Reviewers get fewer misleading comments, evidence they can inspect for each confirmed defect, and a clear next action when Bugbot cannot establish a verdict. Repository maintainers can keep reliable merge gates while distinguishing an incomplete review from both a clean review and a workflow failure. The feature applies consistently across languages and frameworks.

Pilot before enabling publication

1. Freeze at least three historical case pairs with pinned source, relevant surrounding files and versioned or executed evidence. Use 🐛 - Bugbot reports valid supplementary Unicode as rejected by a Unicode aware regular expression #898 for Unicode-aware matching, 🐛 - Bugbot reports defaulted pytest arguments as missing fixtures #914 for fixture discovery with defaulted arguments, and 🐛 - Bugbot misses restored fixed replacement source guard #919 for the earlier SQLite guard. For each historical false finding, author a genuine corresponding defect with a predeclared expected condition and anchor. Examples include a validator that mishandles supplementary characters or accepts malformed UTF-16, a genuinely required unresolved fixture, and a reachable restore/replacement failure when the earlier guard is removed. Keep the expected verdict and scoring metadata outside the model input. Keep evidence from the original reported revision separate from later reproductions.
2. Compare three variants using GPT-6 Luna and GPT-6.1 Sol. Use the existing flow as the baseline, a general second pass with the detection context as the extra-inference control, and a general second pass with application-retrieved project evidence as the proposed flow. Keep the source, task scope, effort, tool permissions and evidence conditions comparable within each model. Record exact SDK, provider, model, effort, runner and fixture identities. Record any unavailable model or unavoidable configuration difference explicitly.
3. First replay the same frozen eligible candidates through the verification variants to isolate their effect. Then run the full detection-to-result path to measure missed real defects and final output. Predeclare three repetitions per matrix cell and retain failures, aborts, divergences and inconclusive results. Measure unsupported active findings, confirmed real defects, missed defects, inconclusive rate, evidence requests, latency and provider usage or cost where available. Report per-case and per-model results and aggregate counts. Use an isolated dry run with zero GitHub review, status-card, check, label, approval or finding-state writes. Reuse the existing evaluator where it applies and add a repository-shaped runner for cross-file evidence. A diff-only benchmark cannot qualify the new behavior by itself.
4. The qualification must demonstrate fewer unsupported published findings in a reproducible paired comparison and preserve the three genuine defect controls and the existing applicable recall floors. Report false positives and recall together. Set explicit evidence, latency and usage budgets before production enablement and report observed overhead against those budgets. If the baseline does not reproduce enough false findings to demonstrate a reduction, report that limitation, expand the frozen cohort under the same scoring rules and avoid claiming an improvement from that cohort. A historical replay, a deterministic test and a real model trial are separate evidence.

Acceptance criteria

• A non-Python eligible finding and a Python eligible finding take the same verification path. Evidence from unchanged files can refute either. A positive control that removes the decisive guard or violates the actual versioned contract remains actionable and correctly anchored.
• Every eligible new, updated, reopened and overflow candidate has exactly one verdict bound to the current review and to evidence actually supplied by the application. Confidence, a bare link and an unexecuted test claim cannot substitute for that evidence. A failed replacement check preserves the confirmed finding as prose and prevents an unverified executable suggestion.
• Evidence from a different commit, unrelated CI job, wrong resolved dependency, unsafe path, forged locator, stale or replayed receipt, or unauthorized execution cannot support publication. Missing material evidence yields an inconclusive result with a named next step. A required missing or malformed verifier result fails the run before new finding mutations.
• A completed run containing confirmed, refuted and inconclusive candidates publishes the confirmed findings in one coordinated result, omits refuted and inconclusive inline defects, and shows the separate verification status. Inconclusive review evidence prevents an automatic clean approval. With only refuted candidates, an otherwise complete clean review is possible under the existing approval rules. Provider publication failures report observed effects and remain recoverable.
• The canonical PR, framed and ordinary paths, the supported Commit fallback and an exact-head recheck share the verification rule. Push and PR paths cannot duplicate publication. Cancellation, supersession, invalid integrity evidence and aggregate budget failure before publication preserve the last verified result and historical findings. A fresh retry for the same revision is idempotent and a newer revision wins.
• A historical finding changes to fixed, obsolete or reopened only with explicit verified evidence, its retained identity and the existing native thread rules. Omission, uncertainty and verifier failure preserve state. A protected human dismissal remains effective. Pending, complete, incomplete, failed and superseded states are legible in the existing GitHub surfaces and localization path.
• The pilot has a reproducible manifest, paired controls, frozen run protocol, raw outcome counts and a reviewable decision on verifier configuration and rollout limits. The SDD maps each acceptance requirement to automated evidence. Add at least 30 distinct deterministic cases, separate from model trials, allocated as 8 provenance and source cases, 6 verdict and publication cases, 6 lifecycle and concurrency cases, 5 permission and security cases, and 5 presentation and compatibility cases. Preserve applicable repository coverage and CI gates. Exercise the production orchestration with a controlled GitHub provider and failure injection, in addition to pure policy tests.
• Update the owning SDDs and catalog evidence, Bugbot detection and finding-publication docs, operator recovery and quality documentation. Cover evidence prerequisites, rollout, limits, historical compatibility, and rollback using the approved runtime route. Validate the specification and generated catalog as applicable, run the affected test and CI suites, and provide the final-head qualification evidence in the PR.

Alternatives considered

The pilot treats a model change and an extra call on the same context as controls. Repository-wide input expansion could improve discovery, but it does not identify the material evidence supporting a particular finding and makes review cost harder to bound. Language-specific evidence adapters remain useful underneath a common verification contract. Publishing all candidates as actionable findings would expose unresolved hypotheses to users and approval consumers. Holding every confirmed finding until all hypotheses are settled would delay useful feedback when the run is otherwise complete. The proposed product flow publishes only confirmed findings and makes uncertainty visible in the shared review result.

Version of copilot

Audited baseline v3.10.13. Feature applies to the supported Bugbot review workflow.

Additional context or comments

Affected reports and regression map

These are the 27 finding-quality reports identified in the audit. They are linked as concrete inputs to this feature. Record for each report the decisive evidence, expected disposition, and the regression or remaining limitation that covers it. Keep the original reports available for case-specific validation and handle their closure based on evidence for each report.

Repository behavior, guards, and project context (14)

• #859 — Follow the imported schema to the existing unique index before claiming a composite foreign key is invalid.
• #861 — Resolve the migration guide in the reviewed Git tree before reporting a missing link target.
• #872 — Follow the UI state transition and distinguish a rendered confirmation value from an editable input.
• #879 — Trace inherited permissions and distinguish the incorrect default-role premise from any separately supported coupling defect.
• #880 — Read the actual pagination contract and UI wording before interpreting a page replacement as a load-more defect.
• #891 — Trace the measured padding and native click target rather than assuming the click lands on the input.
• #892 — Trace the full event sequence, including the next mousedown that clears suppression.
• #893 — Follow Angular's actual native-reset directive and control-value accessor ordering.
• #896 — Follow the shared listener options to the capture flag. Preserve any separately evidenced shadow-boundary defect.
• #897 — Inspect the rendered light DOM and the native submit-button proxy.
• #902 — Trace the stale-trigger guard and the rendered state at each click.
• #917 — Check the atomic snapshot restore invariant and distinguish retained history from current effects.
• #918 — Follow the current-schedule query and its prefilter for retired schedules before evaluating offered actions.
• #919 — Follow the earlier replacement-source guard and the SQLite query over retained fixed exceptions.

Language, browser, and toolchain behavior (13)

• #867 — Check the resolved Playwright matcher contract for the supported enabled option.
• #869 — Trace the proxied readiness route and the pinned Playwright HTTP readiness predicate.
• #873 — Check the rendered Markdown before treating fenced sample links as live navigation.
• #875 — Inspect the actual JSON value types and exact-integer guard.
• #888 — Distinguish native requestSubmit and reset from click activation and preserve the genuine duplicate-click case.
• #890 — Check native implicit submission and the role of disabled text inputs in the blocker count.
• #894 — Use the pinned Vitest Browser matcher semantics for aria-disabled.
• #895 — Inspect resolved Vitest and Storybook collection and coverage settings.
• #898 — Verify supplementary Unicode matching and retain detection of malformed UTF-16.
• #900 — Verify FormData with a null submitter and the form's actual remaining controls.
• #904 — Check the exact compiler version and relevant TypeScript job before claiming a type failure.
• #913 — Evaluate the complete fractional pattern and the concrete six-digit witness.
• #914 — Check fixture discovery in the installed pytest release and compare against a genuinely required missing fixture.

Implementation handoff

Follow the existing feature admission and pre-branch SDD flow. Use the existing Bugbot analysis and publication capability and coordinate with finding-state reconciliation and framed review. Record the decisions for evidence profiles and limits, execution authorization, mixed publication and approval behavior, and rollout in the SDD. Keep pure admission and disposition policies separate from source and execution adapters, evidence orchestration, provider mutations and presentation. Add an executable boundary check so verifier ports and untrusted response handling cannot invoke publication or arbitrary commands. Keep the pilot separate from activation and preserve the approved release and rollout process.

On-demand evidence revision decisions

• The first verifier response is a final verdict or one typed request. The application reads only the authorized repository and revision within the candidate budget and returns a candidate-bound receipt with the fragment. The follow-up is final and cites its actual evidence. Missing, denied or exhausted decisive evidence remains inconclusive with cause and next step. Invalid citations stop finding, history and approval writes for the entire batch.
• Before freezing identity, retain a valid substantive anchor. A syntactic separator or non-addressable line may move to a unique eligible causal line within two lines of the same change block. Record the original and selected positions. Ambiguity withholds an inline location. The verifier cannot move the anchor. Deterministic tests include the pytest separator, an eligible line to preserve and two plausible lines.
• V4 and the Sol supplement are diagnostic and v4 did not qualify. Compare Luna and Sol at low effort in the replay, using at most 90 seconds, 32,768 gross input and 4,096 output tokens per case including both rounds and retries. A favorable held-out replay would allow full-review evaluation. Require all existing historical reproduction, paired improvement, positive control, exact-anchor, integrity, cost, latency and independent 174-case detection gates for each exact profile independently. Activation remains disabled and requires explicit maintainer approval after qualification.

Bugbot verification integration and rollout for #950

1. Executive summary

Bugbot detects candidates, collects applicable project and official evidence,
independently verifies each candidate, and coordinates the result before any
publication. This amendment records the current delivery and rollout decisions.
The existing owner retains the historical cases and earlier pilot evidence.

4. Goals

6. Functional behavior and state model

A review prepares its candidates before verification. The normal reviewer
configuration is inherited unless the repository selects a compatible verifier
profile. Open a fresh session for each candidate and supply its hypothesis,
anchor and pinned source. Follow the effective implementation, configuration
and dependency versions. Consult applicable official versioned sources or CI
when decisive. Use a typed, read-only evidence broker with provenance and
explicit missing or ambiguous results. The same investigation may refine an
incomplete citation before closing. Time and token use are telemetry, with
explicit cancellation and provider failures recorded separately.

Each candidate gets a confirmed, refuted or inconclusive verdict. The closure
binds material claims to delivered source ranges and preserves the anchor.
A complete mixed review can publish confirmed findings when enforcement is
active. An inconclusive review does not authorize a clean approval. An
integrity failure, cancel or moved head preserves the prior verified result.
In shadow, record the outcome separately from ordinary publication. The
default remains disabled. Enforcement requires a qualified exact profile and
explicit maintainer approval. Approval uses current review and policy evidence;
a previous receipt cannot authorize a new policy or rollback generation.

8. Clean Architecture

Application use cases own sequencing, source identity, verdict completeness,
publication and approval. Read-only adapters acquire Git, official and CI
evidence. Provider adapters perform investigation and return bounded structured
verdicts. GitHub adapters write only through the publication boundary. The
release decision remains with the maintainer. The source CI and review checks
remain independent of feature activation.

14. Testing strategy

15. Documentation

16. Acceptance

Validate the source, exact anchor, verdict and policy bindings in the ordinary
and framed routes. Keep the 27 historical reports traceable and keep recorded
pilots reproducible. Each profile proposed for enforcement must independently
satisfy the existing historical reproduction, paired improvement, positive
control, integrity and latency criteria and the independent 174-case detection
corpus. Evidence for readiness and authorization for production are distinct.

17. Requirements traceability

20. References and decisions

The existing Bugbot SDD and its evidence reports document the qualification
thresholds, supported profiles, approval policy and rollback protocol.
This revision neither enables production verification nor qualifies a profile.