From b3772da33f66a58bf8fb86af66783a1b3e537600 Mon Sep 17 00:00:00 2001 From: Xinyuan Lin Date: Wed, 12 Aug 2026 21:18:14 -0700 Subject: [PATCH 1/3] fix(test, frontend): raise Vitest timeouts for CI stalls The macOS leg of `build / frontend` goes red on a different unit test every few days -- always a timeout, never the same spec, always green on rerun. Three occurrences in the last four days: | Run | Test | Error | | --- | --- | --- | | 31665399757 | UserDatasetVersionCreatorComponent > onClickCreate ... | Test timed out in 5000ms | | 31630884042 | AdminUserComponent > sortByAffiliation ... | Hook timed out in 10000ms | | 31411656559 | WorkflowRuntimeStatisticsComponent > should create | Test timed out in 5000ms | The tests are not the problem: the runner stalls, and the stall lands on whichever test is executing. In run 31665399757 the offending spec file took 11727ms on macos-latest and 240ms on ubuntu-latest for the same commit; in an earlier run the same file took 219ms on macOS. Suite totals from that run show the same picture -- 252.88s wall on macOS vs 89.85s on ubuntu, with a cumulative test time of 307.69s vs 182.34s. macos-latest gives 3 cores and 7 GB against ubuntu's 4 and 16, so the jsdom + v8-coverage workers run under real memory pressure there. Raise testTimeout and hookTimeout to 30s in both Vitest configs, which absorbs a stall an order of magnitude worse than any observed so far. A spec that legitimately needs 30s is broken, and the job's own timeout still bounds a true hang. Per-test timeouts would be whack-a-mole: the next stall picks a different test. Also opt the frontend matrix out of fail-fast, as every other multi-leg matrix in build.yml already does. Today one flaky OS cancels the other two legs, which destroys exactly the evidence needed to tell a runner flake from a real break. Before: macOS stalls 5s -> that test fails -> ubuntu + windows cancelled After: macOS stalls 5s -> absorbed; a real break still fails all legs --- .github/workflows/build.yml | 6 ++++++ frontend/TESTING.md | 1 + frontend/vitest.browser.config.ts | 5 +++++ frontend/vitest.config.ts | 10 ++++++++++ 4 files changed, 22 insertions(+) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index c28457b1986..826fd12ef7b 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -87,6 +87,12 @@ jobs: if: ${{ inputs.run_frontend }} runs-on: ${{ matrix.os }} strategy: + # An OS-specific failure should not cancel the other two legs: with the + # default fail-fast the surviving jobs report "The operation was + # canceled" and the run no longer says whether the failure reproduces + # off that OS — exactly the evidence needed to tell a runner flake from + # a real break. Every other multi-leg matrix here already opts out. + fail-fast: false matrix: os: [ubuntu-latest, windows-latest, macos-latest] include: diff --git a/frontend/TESTING.md b/frontend/TESTING.md index 7372fc4b29d..9e4412d7e70 100644 --- a/frontend/TESTING.md +++ b/frontend/TESTING.md @@ -47,6 +47,7 @@ For repo-wide testing philosophy (TDD, characterization tests, "every test must | Coverage | `@vitest/coverage-v8` | | Test setup | `src/test-zone-setup.ts` wraps `it`/`test` in an Angular ProxyZone (Vitest does not provide one and Angular's `fakeAsync` requires it) | | Globals | `globals: true` in `vitest.config.ts`, so `describe / it / expect / vi / beforeEach` come from the runtime — no per-file imports | +| Timeouts | 30s per test and per hook, raised from the 5s/10s Vitest defaults because macOS CI runners stall for seconds at a time (#6073) | `src/main.test.ts` is intentionally a near-empty `export {}`. The `unit-test` builder uses `buildTarget`'s `main` to seed the bundle graph; if it pointed at the real `main.ts`, every component declared in `AppModule` would be type-checked for every spec, surfacing template errors for components no active spec touches. Keeping `main.test.ts` empty narrows the graph to what each spec actually imports. diff --git a/frontend/vitest.browser.config.ts b/frontend/vitest.browser.config.ts index 3fe2c410390..b67414aa392 100644 --- a/frontend/vitest.browser.config.ts +++ b/frontend/vitest.browser.config.ts @@ -64,6 +64,11 @@ export default defineConfig({ // browser-mode the runtime has neither, so we install the `buffer` npm // package as a shim). setupFiles: ["src/browser-buffer-polyfill.ts", "src/test-zone-setup.ts"], + // Same runner-stall headroom as the jsdom config (vitest.config.ts): + // driving a real Chromium through playwright is strictly slower than + // jsdom, so these specs need at least as much slack. See #6073. + testTimeout: 30_000, + hookTimeout: 30_000, browser: { enabled: true, provider: playwright(), diff --git a/frontend/vitest.config.ts b/frontend/vitest.config.ts index 9cb2f82f88c..dbfc9a78295 100644 --- a/frontend/vitest.config.ts +++ b/frontend/vitest.config.ts @@ -34,6 +34,16 @@ export default defineConfig({ // which Angular's `fakeAsync` requires. Karma+Jasmine installed this // implicitly; the @angular/build:unit-test path doesn't. setupFiles: ["src/test-zone-setup.ts"], + // Vitest defaults (5s per test, 10s per hook) are too tight for the + // macOS runners, which stall for seconds at a time under load: the same + // spec file that takes 240ms on ubuntu-latest has been observed taking + // 11.7s on macos-latest in the same commit's matrix. The stall lands on + // whichever test happens to be running, so raising the ceiling is the + // only fix that isn't whack-a-mole — three different specs have gone + // red this way. A test that legitimately needs >30s is broken, and the + // job's own timeout still bounds a true hang. See apache/texera#6073. + testTimeout: 30_000, + hookTimeout: 30_000, // Per-spec exclusions live in `angular.json` (the unit-test builder // applies them at the discovery stage, before Vitest's own filter, // which is what the Vitest team recommends — see the Vite warning From fb5ec5a4ad0a574a01769b7d381b43f0a437f84e Mon Sep 17 00:00:00 2001 From: Xinyuan Lin Date: Fri, 14 Aug 2026 23:35:14 -0700 Subject: [PATCH 2/3] fix(test, frontend): bound the frontend job and scope the timeout notes Review follow-ups: the frontend job had no `timeout-minutes`, so the claim that "the job's own timeout bounds a true hang" rested on GitHub's implicit 6h cap; browser mode already resolves a 30s `hookTimeout` from `browser.enabled`, so setting it there restated the default; and the fail-fast rationale named "every other multi-leg matrix", which `amber-integration` and `pyamber` refute. --- .github/workflows/build.yml | 9 ++++++++- frontend/TESTING.md | 2 +- frontend/vitest.browser.config.ts | 10 ++++++---- frontend/vitest.config.ts | 3 ++- 4 files changed, 17 insertions(+), 7 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 826fd12ef7b..48e9f831638 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -86,12 +86,19 @@ jobs: frontend: if: ${{ inputs.run_frontend }} runs-on: ${{ matrix.os }} + # Only the "Install dependency" step below is bounded, so a spec that + # truly hangs would otherwise run to GitHub's implicit 6h cap — billed at + # 10x on the macOS leg. Recent green runs take 4-12 minutes and install + # alone may take 20, so 30 bounds a hang while still absorbing a cold + # yarn cache on a slow runner. + timeout-minutes: 30 strategy: # An OS-specific failure should not cancel the other two legs: with the # default fail-fast the surviving jobs report "The operation was # canceled" and the run no longer says whether the failure reproduces # off that OS — exactly the evidence needed to tell a runner flake from - # a real break. Every other multi-leg matrix here already opts out. + # a real break. `platform`, `platform-integration`, `agent-service` and + # `infra` opt out for the same reason. fail-fast: false matrix: os: [ubuntu-latest, windows-latest, macos-latest] diff --git a/frontend/TESTING.md b/frontend/TESTING.md index 9e4412d7e70..e27b86d09e3 100644 --- a/frontend/TESTING.md +++ b/frontend/TESTING.md @@ -47,7 +47,7 @@ For repo-wide testing philosophy (TDD, characterization tests, "every test must | Coverage | `@vitest/coverage-v8` | | Test setup | `src/test-zone-setup.ts` wraps `it`/`test` in an Angular ProxyZone (Vitest does not provide one and Angular's `fakeAsync` requires it) | | Globals | `globals: true` in `vitest.config.ts`, so `describe / it / expect / vi / beforeEach` come from the runtime — no per-file imports | -| Timeouts | 30s per test and per hook, raised from the 5s/10s Vitest defaults because macOS CI runners stall for seconds at a time (#6073) | +| Timeouts | 30s per test in both configs — a raise from Vitest's 5s jsdom default, and from the 15s it resolves under `browser.enabled`. Hooks get 30s under jsdom (up from 10s); browser mode already defaults to 30s. macOS CI runners stall for seconds at a time (#6073) | `src/main.test.ts` is intentionally a near-empty `export {}`. The `unit-test` builder uses `buildTarget`'s `main` to seed the bundle graph; if it pointed at the real `main.ts`, every component declared in `AppModule` would be type-checked for every spec, surfacing template errors for components no active spec touches. Keeping `main.test.ts` empty narrows the graph to what each spec actually imports. diff --git a/frontend/vitest.browser.config.ts b/frontend/vitest.browser.config.ts index b67414aa392..7e8368bb77a 100644 --- a/frontend/vitest.browser.config.ts +++ b/frontend/vitest.browser.config.ts @@ -64,11 +64,13 @@ export default defineConfig({ // browser-mode the runtime has neither, so we install the `buffer` npm // package as a shim). setupFiles: ["src/browser-buffer-polyfill.ts", "src/test-zone-setup.ts"], - // Same runner-stall headroom as the jsdom config (vitest.config.ts): - // driving a real Chromium through playwright is strictly slower than - // jsdom, so these specs need at least as much slack. See #6073. + // Browser mode already resolves larger defaults than jsdom (15s per test, + // 30s per hook, keyed off `browser.enabled`), but driving a real Chromium + // through playwright is strictly slower than jsdom, so the per-test + // ceiling is lifted to the same 30s the jsdom config uses + // (vitest.config.ts). `hookTimeout` is left alone — its browser-mode + // default is already 30s. See #6073. testTimeout: 30_000, - hookTimeout: 30_000, browser: { enabled: true, provider: playwright(), diff --git a/frontend/vitest.config.ts b/frontend/vitest.config.ts index dbfc9a78295..f322177aa0d 100644 --- a/frontend/vitest.config.ts +++ b/frontend/vitest.config.ts @@ -41,7 +41,8 @@ export default defineConfig({ // whichever test happens to be running, so raising the ceiling is the // only fix that isn't whack-a-mole — three different specs have gone // red this way. A test that legitimately needs >30s is broken, and the - // job's own timeout still bounds a true hang. See apache/texera#6073. + // frontend job's `timeout-minutes: 30` (.github/workflows/build.yml) + // still bounds a true hang. See apache/texera#6073. testTimeout: 30_000, hookTimeout: 30_000, // Per-spec exclusions live in `angular.json` (the unit-test builder From 2e6870329c9b595f3484717bfa939675170858c6 Mon Sep 17 00:00:00 2001 From: Xinyuan Lin Date: Sun, 16 Aug 2026 19:13:30 -0700 Subject: [PATCH 3/3] fix(test, frontend): size the job cap from observed run times The justification for `timeout-minutes: 30` cited the "Install dependency" step's own 20-minute budget as though it were an observed duration. It isn't: install measures 37s / 49s / 74s on ubuntu / macOS / windows. The conclusion did not follow either -- had install actually approached 20, the remaining steps still need 8-9.5 minutes, putting the job at 28-29.5 against the cap, which is the opposite of the absorption the sentence claimed. Size it from the legs instead. From this PR's run 31869822969: | Leg | Job | Install | Rest | | --- | --- | --- | --- | | ubuntu-latest | 8.6 min | 37s | 8.0 min | | macos-latest | 10.0 min | 49s | 9.2 min | | windows-latest | 10.7 min | 74s | 9.5 min | The value stays at 30 -- ~3x the slowest leg, ~19 minutes of slack. --- .github/workflows/build.yml | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 48e9f831638..8c1659aad27 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -88,9 +88,10 @@ jobs: runs-on: ${{ matrix.os }} # Only the "Install dependency" step below is bounded, so a spec that # truly hangs would otherwise run to GitHub's implicit 6h cap — billed at - # 10x on the macOS leg. Recent green runs take 4-12 minutes and install - # alone may take 20, so 30 bounds a hang while still absorbing a cold - # yarn cache on a slow runner. + # 10x on the macOS leg. Sized off observed green legs, which run 8.6-10.7 + # minutes end to end (install itself lands under 90s on all three OSes): + # 30 is ~3x the slowest, leaving ~19 minutes of slack for a cold yarn + # cache without masking a hang. timeout-minutes: 30 strategy: # An OS-specific failure should not cancel the other two legs: with the