Skip to content

ci(pr-verify): add the capture job that drives the affected app (S7 of #8651) #12232

Open

Description

@kyle-sexton

Parent

Refs #8651. Phase S7 ("Phase 10: S7 pr-verify capture job") of the evidence-verification plan, durable copy at #11364 (comment); Brief at #11364 (comment). #11393 (S5c) waits on this job; before this issue existed it was blocked by #8651 itself, its own parent.

Relates to #7529 (Claude Code UI leg of the per-PR smoke).

Depends on

Outcome

pr-verify runs a capture job on in-repo PR heads. The job drives the affected app unattended, uploads the publishable captures as a retained artifact, and feeds them to the judge jobs as untrusted data. A missing or blocked capture never fails a verdict.

Plan (phase S7)

Satisfies: AC1, AC2, AC5, AC9, AC10
Size: Large. Files: .github/workflows/pr-verify.yml, .github/scripts/pr-verify/prompt.md, .github/scripts/pr-verify/workflow.test.mjs. Design: T8, sequence flow 2.

  • Probe step first. After setup-runner-isolation, a head-code step tries to read the runtime token and the credential from a hook. The job fails closed if either is readable.
  • Job capture. It uses the same if as the existing judge jobs, so fork heads never run. It runs the gates and uses a read-only token, .base plugins and --trusted-root .base, then calls /testing:run-e2e unattended app=<merge.verify-capture.app> base=<base-sha>. It uploads the publishable files as pr-capture-<pr>-<sha> with retention-days. When the key is unset, it skips with a note.
  • Gates first. The capture job's first steps are the kill switch, resolver, trusted-trigger and agree steps (check-intent's copies stay), so it captures nothing on a run that judges nothing. It sets timeout-minutes: 30. The outer session runs through ci-workflows' run-claude-cli with its own role, a tool allowlist and the drive maxEntryPoints cap (S4b). A capture timeout leaves capture: blocked and never stops the judge.
  • Judge jobs. Only check-intent gains needs: capture, with !cancelled(). check-intent-subscription and check-intent-fallback keep their success chain and download the same run's artifact by id. All three download into pr-verify-data/capture/. The prompt frames the capture as untrusted data and says a missing or blocked capture is never a FAIL reason. A trusted-text manifest line Evidence loop: 3/3 cap reached, last NEEDS WORK is a FAIL (AC7's park path then runs).
  • Last step. After the probe passes on a real PR, set merge.verify-capture.app: claude-code and show one in-repo PR with a judged capture (AC1). Update the header comment block in pr-verify.yml that describes the jobs (reviewed change: jobs now take an artifact).

Credential separation (Q10, Q11)

The superseding Q11 record (#11364 (comment)) is the Issue Triage session's reading of the owner's 2026-10-31 direction. It says:

  • The capture job may hold pr-verify's existing credential (Q10).
  • The PR's own code runs in a child process whose environment and readable files hold no token. The credential stays with the Claude Code driver and is never passed to, exported to, or written where the app under test can read it.
  • In-repo heads only.

If the build cannot keep the token out of the PR code's reach, it stops and brings that back to the owner as a new question. It does not ship without the separation.

Done when

  • node --test .github/scripts/pr-verify/workflow.test.mjs exits 0 with these cases:
    • the capture job's permissions contain no write;
    • the three judge jobs carry identical capture-download steps;
    • the capture if includes the head-repository check;
    • pr-verify.yml has no job-level always();
    • only check-intent has needs: capture;
    • a probe step exists, runs after setup-runner-isolation and before any head code, and has no continue-on-error.
  • The capture job's step that starts the app under test passes it no credential variable, and a test asserts that the app process's environment holds none of the pr-verify credential names.
  • actionlint passes on pr-verify.yml.
  • One in-repo PR shows a judged capture with merge.verify-capture.app: claude-code set (AC1).

Builder: the Automation Lanes session. This is host-run work: it edits a workflow file and needs a live PR run.

Metadata

Metadata

Assignees

No one assigned

    Labels

    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.priority: mediumReal value, no hard deadline; normal backlog flow.status: readyTriaged, unblocked, and fully specified; eligible to pick up.work-class: structuralRefactors, migrations, contract changes; cross-cutting and hard to reverse.

    Type

    Fields

    No fields configured for Task.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions