feat(wizard-ci): real-TUI e2e + snapshot review#2012
Open
gewenyu99 wants to merge 32 commits into
Open
Conversation
gewenyu99
changed the title
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Adds two modes to the existing wizard-ci, as an alternative to classic --ci (LoggingUI: agent-only, stdout-grep). --e2e drives the WHOLE interactive flow headlessly through the wizard-ci-tools control plane and asserts on structured state; --replay plays a recorded run back in the terminal. Core files: - services/wizard-ci/e2e.ts — runE2e(): /tmp app-copy isolation, env hygiene (strips host CLAUDE*/ANTHROPIC* so the spawned agent auths with the phx key instead of deferring to the host), scoped --project-id, the happy-path policy (skip mcp+slack, delete skills, continue past health issues), spawns the wizard repo's headless harness, then asserts the structured result (runPhase=completed, posthog dep/.env, reached keep-skills, skillsComplete). replayRecording(): shells to the wizard repo's terminal replayer. - services/wizard-ci/index.ts — wires --e2e (positional app, --project-id, --keep-skills) and --replay (--step/--delay) into the CLI + --help. Engine lives in the wizard repo (store + driver must run in-process); point WIZARD_PATH at it. See PostHog/wizard PR for src/lib/ci-driver + harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nitions Run each CI-e2e test definition (for now: integration on express-todo) as a real --e2e agent run, render every key-moment frame of the recording to a real-Ink ANSI snapshot, and diff against a committed baseline. Surfaces run-to-run differences (e.g. the agent enqueuing tasks differently) side-by-side for a human to review — same screens every run, deltas flagged. No mocks: real agent, real recording, real render. - services/wizard-ci/snapshots.ts — the flow (run → render → diff → report) - services/wizard-ci/ansi-html.ts — dependency-free ANSI→HTML for the side-by-side - services/wizard-ci/snapshots/express-todo/ — committed baseline (47 frames) - pnpm wizard-ci-snapshots (+ mprocs entry); --update to accept a new baseline Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The snapshots.ts header now lists what the flow needs in .env (POSTHOG_PERSONAL_API_KEY, POSTHOG_WIZARD_PROJECT_ID, POSTHOG_REGION) and that WIZARD_PATH must point at a checkout containing e2e-harness/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A real agent emits frames a little differently run to run (different number of status updates → shifted indices), so drift is expected. Print the per-frame diffs + report.html and exit 0; only a genuine failure (run died, no recording) exits non-zero. Accept a new baseline with --update. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
After the diff, prompt "Replay <name> snapshots in the terminal? [y/N]" and, on yes, launch the replay stepper directly on the run's recording — no copy/paste. TTY-only (auto-declines in CI so nothing hangs); the replayer inherits stdio for its own Enter-to-step loop. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Document handing the Wizard to an agent to run/drive/explore it headlessly, pointing at the runbook (wizard repo e2e-harness/EXPLORING-AS-AN-AGENT.md) with a copy-paste example prompt that targets wasp-lang/open-saas — the agent works out how to build + run the target itself. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… comments The agentic-exploration section belongs in the wizard repo's README, not here. Also trim snapshots.ts / index.ts comments to concise current-behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…wright) services/wizard-ci/screenshot.ts rasterizes the side-by-side report — one PNG per key-moment frame (baseline │ current) plus a full-flow strip — for attaching to a review PR. Reuses the report HTML (ansi-html), so no new ANSI logic. Adds playwright as a dev dep. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…shots snapshot-review.ts runs the e2e, renders the report to side-by-side PNGs, and opens a review PR whose body embeds them (raw URLs), changed frames first — instead of running the agent evaluator. --dry-run writes the bundle locally. wizard-snapshots.yml dispatches it (bot token, setup-wizard-deps, Playwright). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`wizard-ci --e2e` and `wizard-ci-snapshots` run the wizard repo's tui-snapshots: the real wizard TUI, driven by store state manipulation, captured per screen as text. --e2e asserts on the result JSON it emits; snapshots diff the captured screens against a committed baseline; snapshot-review rasterizes them to a side-by-side image PR. Drops the recording/replay plumbing (the --replay flag, the render step, ansi-html) — the captured screens are already clean text. WIZARD_PATH defaults to a sibling wizard checkout. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…as a comment Comment `/wizard-ci [app] [wizard_ref]` on a PR to run the real-TUI e2e. The workflow acks with 👀, checks out the PR, runs snapshot-review, and posts a comment on the PR with the flow strip and a link to the full side-by-side review (--comment-pr). Restricted to repo members/owner/collaborators. Manual workflow_dispatch still works; no auto-run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rue) A standalone snapshots job, independent of the evaluator (the eval still runs as normal). Dispatch "Wizard CI" with snapshots=true to also open a real-TUI review PR for the app — same app token + setup-wizard-deps + PostHog key as the evaluator, project hard-coded to 2 (the bot key's project). Because wizard-ci.yml is on main, this is dispatchable from a PR branch (pre-merge). The /wizard-ci comment trigger stays in wizard-snapshots.yml. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…snapshot) The wizard-ci job's Execute step runs the headless eval by default, or — with the snapshots=true input — the real-TUI snapshot review for the app. One switch, the same job; no separate parallel job. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…sterize) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…R step)
git('rev-parse HEAD'), not the array form — the helper runs git ${cmd}. Pass cwd
to getRepoRoot too. tsc clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…able text) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Match the eval path's title and body conventions (buildPRTitle/buildPRBody): [CI] (snapshots) <app> title, plain metadata lines, no marketing prose. Drop the unreadable full-flow composite (_flow.png) from the PR body, the comment, and screenshot.ts; the per-frame side-by-side images remain. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Align the snapshot path's project id with the evaluator's convention (secrets.GH_APP_POSTHOG_WIZARD_CI_BOT_TARGET_PROJECT_ID) instead of the '2' placeholder, and drop the now-stale _flow.png mentions from the header. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
main (#2042) already sets POSTHOG_WIZARD_PROJECT_ID from the bot's target-project secret on the Execute step. Keep that one; this branch adds only the snapshots switch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gewenyu99
force-pushed
the
wizard-e2e-control-plane
branch
from
86e2813 to
1bdfbed
Compare
Replace the explicit strip list with the same /^(CLAUDE|ANTHROPIC|AI_AGENT)/ predicate the wizard's MCP host uses, so the two can't drift. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…OCTOU' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
The issue_comment trigger checked out and ran the PR's code with the bot secret and write perms — CodeQL untrusted-checkout-toctou (7 critical). The Copilot autofix only pinned the ref (still runs untrusted code, and issue_comment payloads have no pull_request.head.sha so it fell back to the merge ref). Use the existing comment mechanism instead: the trusted handler parses the comment and fires a repository_dispatch; this workflow runs only on dispatch and checks out its own trusted ref. comment_pr in the payload still drives the post-back. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
semgrep run-shell-injection: event-controlled values were interpolated straight into the run script. Bind them to env vars and reference shell variables instead, matching wizard-ci.yml's resolve-inputs step. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Keep the PR small: remove the committed baselines and the baseline-vs-current comparison. The snapshot run now captures the real-TUI key-moment frames and surfaces them (report.html + a review PR with one image per frame); regression comparison is out of scope for now. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… set snapshot-review forwards its app to snapshots.ts, which ignored it and always ran express-todo. Honor a positional app arg so per-app CI (matrix / dispatch) snapshots the requested app. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…workflow The snapshots switch only fired on workflow_dispatch (inputs.snapshots is empty for repository_dispatch), so the comment path needed its own workflow. Have SNAPSHOTS also read client_payload.snapshots, so the /wizard-snapshots comment routes through the same wizard-ci-trigger dispatch and the one 'if SNAPSHOTS' branch. Removes wizard-snapshots.yml. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…vices Resolve wizardRepo() once per run; hoist OUT_ROOT and the Shot type into e2e.ts and import them in snapshots/snapshot-review/screenshot instead of re-declaring. Fix a stale doc reference (/wizard-ci -> /wizard-snapshots). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
snapshots.ts (writer) and snapshot-review.ts (reader) each rebuilt join(OUT_ROOT, name, ...) independently — a path contract that could drift. Hoist reportDirFor(app) into e2e.ts and use it on both sides. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Derive each frame's seconds-since-the-first-frame from its write time (frameTimings) and append it to the report headings and the PR-body headings (e.g. '23-outro.txt (+1m23s)'), so a reviewer can see how long each key moment took to reach. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gewenyu99
commented
Jun 25, 2026
Contributor
Author
There was a problem hiding this comment.
e2e.ts: runs the snapshotting path, for local run through mprocs and for CIsnapshots.ts: local viewing, zero external deps.screenshot.ts: isolates the Chromium dep; standalone. this is to make it easy to read as comments in CIsnapshot-review.ts: isolates the GitHub side; CI-only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Usage
All four drive the real wizard TUI and surface the current run's key-moment frames (no baseline comparison).
1. Local — run + HTML report
Runs the real agent flow against express-todo through the real wizard TUI, captures each key moment, writes
report.htmlof the current frames (WIZARD_PATHauto-resolves to a siblingwizard-e2e; creds from.env).2. Local — review bundle (no PR)
3. Manual dispatch (CI) — the
snapshotsswitch picks which CI runs; the normal evaluator is the default and is unchanged:Opens one review PR per app.
Example output — the next.js matrix from this PR:
4. PR comment — distinct commands pick the mode (members, via the bot):
Both go through the same
wizard-ci-triggerrepository_dispatch and the singleSNAPSHOTSbranch inwizard-ci.yml;/wizard-snapshotsjust addssnapshots: trueto the payload (handled in PostHog/wizard#728). The handler only parses + dispatches — no untrusted PR code is checked out. Requires both this PR and PostHog/wizard#728 merged (repository_dispatch only reaches workflows on the default branch).What this is
wizard-ci --e2eandwizard-ci-snapshotsdrive the real wizard TUI (via the wizard repo'stui-snapshots): the realstartTUI, driven by state manipulation, captured per key moment as text.--e2easserts on the result JSON the run emits (run completed, posthog dep /.env, reachedkeep-skills).wizard-ci-snapshotssurfaces the current run's real-TUI key-moment frames asreport.html(no baseline comparison for now).snapshot-reviewrasterizes the frames to an image PR (one frame per row, titled[CI] (snapshots) <app>), and posts the report back as a comment when triggered with acomment_pr.The snapshot path reuses the existing Wizard CI workflow: one
snapshotsinput switches the Execute step from the evaluator to the real-TUI review.Companion PRs: PostHog/wizard#702 (the harness, merged) · PostHog/wizard#728 (the
/wizard-snapshotscomment trigger).