Skip to content

Define merge semantics for results.errors #93

Open

Description

@jamesarosen

Follow-up to #65, from this comment. Builds on #88, which adds results.errors: [{message, trace, extra}].

Problem

Sharded runs are routinely merged into one CTRF document. Merging is where run-level errors get lost. #88 says nothing about merging. Current CTRF tooling would silently drop errors from every shard but the first:

  • ctrf merge in ctrf-cli keeps extra and environment from the first shard only.
  • ctrf-js merge() keeps the first value for each key and rebuilds the result without any results.errors handling.

So aAshard that failed in global teardown would produce a merged report with no error and, if all its tests passed, a green verdict.

Precedent

Tools that merge run-level errors concatenate them:

De-duplication is also risky: the same message from two shards (for example, "Database cleanup failed") might mean two separate failures.

Proposal

Add to the spec:

When merging CTRF documents (for example, shards of one run), results.errors SHOULD be the concatenation of every input document's results.errors, in input order, without de-duplication. A merging tool MAY record the source document or shard in each entry's extra.

Then update ctrf merge and ctrf-js merge() to follow it.

Related

If an optional run outcome is added (separate issue), its merge rule should follow Playwright's worst-first ordering.

Activity

  1. Ma11hewThomas commented on Nov 14, 2026

    @Ma11hewThomas
    Collaborator

    Thanks, @jamesarosen . PR #90 covers preserving errors from every input in order, without deduplication. Shard attribution you mention is a good suggestion. I’d like to leave it open for now and consider it alongside whether individual errors hold their identity and provenance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions