You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Follow-up to #65, from this comment. Builds on #88, which adds results.errors: [{message, trace, extra}].
Problem
Sharded runs are routinely merged into one CTRF document. Merging is where run-level errors get lost. #88 says nothing about merging. Current CTRF tooling would silently drop errors from every shard but the first:
ctrf-js merge() keeps the first value for each key and rebuilds the result without any results.errors handling.
So aAshard that failed in global teardown would produce a merged report with no error and, if all its tests passed, a green verdict.
Precedent
Tools that merge run-level errors concatenate them:
Playwright's merge-reports replays every shard's onError with no de-duplication. It merges run status worst-first: failed, then timedout, then interrupted, then passed.
De-duplication is also risky: the same message from two shards (for example, "Database cleanup failed") might mean two separate failures.
Proposal
Add to the spec:
When merging CTRF documents (for example, shards of one run), results.errors SHOULD be the concatenation of every input document's results.errors, in input order, without de-duplication. A merging tool MAY record the source document or shard in each entry's extra.
Then update ctrf merge and ctrf-js merge() to follow it.
Related
If an optional run outcome is added (separate issue), its merge rule should follow Playwright's worst-first ordering.
Thanks, @jamesarosen . PR #90 covers preserving errors from every input in order, without deduplication. Shard attribution you mention is a good suggestion. I’d like to leave it open for now and consider it alongside whether individual errors hold their identity and provenance
Activity
Ma11hewThomas commentedon Nov 14, 2026
Thanks, @jamesarosen . PR #90 covers preserving errors from every input in order, without deduplication. Shard attribution you mention is a good suggestion. I’d like to leave it open for now and consider it alongside whether individual errors hold their identity and provenance