Skip to content

Flake audit: shared-fixture + async-import-shim tests masked by CI retry:1 #2813

Closed

Description

@jung-thomas

Why

PR #2812 fixed one non-hermetic unit test (backfill-images-slug-filter.test.ts) that intermittently failed in CI (expected [] to deeply equal ['beta']) under the Vitest-4 forks pool — passing locally, failing only in the full 4-worker run after retry:1 already fired.

The deeper problem it exposed: retry:1 in CI silently masks flaky tests. A test that needs a retry to go green looks healthy until the retry also loses, at which point it reads as 'tests randomly break' and trains the team to ignore red.

Scope (over the coming days — not urgent)

  1. Grep the suite for the same flake shape:
    • per-file beforeAll/afterAll fixtures shared across its (vs per-test beforeEach)
    • modules reached through the workspace-first async-import shim (try { await import('@tutorials/...') } catch { await import(bundle) } — ~116 files use it) imported by unit tests
  2. Surface which tests currently depend on retry:1 to pass — a one-off CI run with retry:0 (or instrumentation) lists every retried-pass; that set is the invisible flake debt.
  3. For each: make it hermetic (per-test fixtures, no shared closure state) + add a loud sanity guard so a future silent empty/undefined fails diagnosably — the pattern applied in test(backfill-images): kill slug-filter CI forks-pool flake #2812.
  4. Decide policy: keep retry:1 (absorbs genuine infra transients) vs scope/annotate it so retried-passes are reported, not hidden.

Reference

Activity

  1. added a commit that references this issue on Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions