feat(cli-exec): add --fail-on-error so CI fails when Percy can't start

[aryanku-dev]

Refs #1181. Related to PER-10368 but NOT the fix for it — see "Provenance" at the bottom.

Problem

percy exec derives its exit code solely from the wrapped command. When Percy itself fails to start — invalid/missing token, browser launch failure — the error is logged and then swallowed, and the pipeline reports success:

[percy] Skipping visual tests
[percy] Error: Invalid API token.
[percy] Running "echo TESTS RAN"
TESTS RAN
[percy] Command "echo TESTS RAN" exited with status: 0
$ echo $?
0

The visual tests silently never ran. This is the long-standing complaint in #1181 (open since 2023).

Root cause

packages/cli-exec/src/exec.js:

• The catch around percy.yield.start() logs Skipping visual tests and continues — the error is discarded.
• The exit code is then taken only from the spawned child: if (status) exit(status, error, false).

So Percy's own error state can never influence the exit code. PERCY_EXIT_WITH_ZERO_ON_ERROR is the opposite knob (it forces 0); there was no way to opt into failing.

Fix

Adds an opt-in --fail-on-error flag (or PERCY_FAIL_ON_ERROR=true) that exits 1 when Percy failed to start.

Default behavior is unchanged — without the flag, the wrapped command's status is still the only input to the exit code, so this is not a breaking change for existing pipelines.

Two deliberate design points:

1. Keyed off the thrown exception, not error-level logs. Scraping logger.query(l => l.level === 'error') looked tempting but would false-positive: percy.js:869 logs Unable to analyze error logs at error level from the error-analysis side channel, and that fires on healthy runs. Verified: a real successful build with --fail-on-error still exits 0.
2. The wrapped command's status wins. If the child exits 3 and Percy also failed to start, the exit code is 3 — the more specific signal.

Scope

This covers startup failures (Percy never started → zero visual coverage). It deliberately does not cover:

• Builds that were created but failed server-side — already covered by percy build:wait.
• Per-snapshot capture failures (e.g. Could not take DOM snapshot). These still produce a finished build with fewer snapshots and exit 0. That is a real remaining gap, called out here so it is not mistaken for covered.

Testing

yarn workspace @percy/cli-exec test — 78/78 pass, including 5 new specs:

• exits non-zero when percy fails to start
• exits non-zero when PERCY_FAIL_ON_ERROR is set without the flag
• exits zero when percy starts successfully
• forwards the command status ahead of a percy start failure
• does not affect the exit code when the flag is not set

Also verified end-to-end against the built CLI:

scenario	exit
invalid token, no flag (default, unchanged)	0
invalid token, --fail-on-error	1
invalid token, PERCY_FAIL_ON_ERROR=true	1
child exits 3 + --fail-on-error	3
valid token, successful build, --fail-on-error	0

Provenance / honest scoping

This started as an investigation into PER-10368 ("Percy errors not failing pipeline"). The ticket's attached screenshot later revealed that customer's actual failure was a @percy/dom bug dropping individual snapshots — fixed separately in #2372 — not a startup failure. So this PR does not resolve PER-10368.

It stands on its own as a fix for #1181, which is a distinct and independently-reported problem. Reviewers should judge it on that basis alone.

Note for reviewers

#1181 was previously answered with "it's by design; we'll evaluate the need for this in the future." This PR keeps that default intact and only adds an opt-in, but the flag name / whether this should eventually become the default is a product call worth confirming.

🤖 Generated with Claude Code