-
Notifications
You must be signed in to change notification settings - Fork 11
2378 lines (2255 loc) · 130 KB
/
Copy pathci.yml
File metadata and controls
2378 lines (2255 loc) · 130 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
name: CI
on:
push:
branches:
- main
# ⭐ NO `branches:` filter here, deliberately (#16482). A `pull_request:`
# trigger filtered to `branches: [main]` does not run at all when the PR's
# base is a feature branch — and a workflow that does not run publishes NO
# check run, which is an ABSENCE, not a skip. Branch protection reads that as
# permanently pending, so every required context this file publishes reports
# nothing on such a PR. Measured on the #14478 stack: six card PRs showed
# 8-11 light checks and ZERO of the six required contexts; the first real
# signal arrived at the trunk PR, after every card was written, reviewed and
# merged. Correctness rested on local runs quoted into PR bodies — a
# discipline, not a gate.
#
# Runner cost of removing the filter, measured 2026-09-08 over the 30 days
# 2026-08-09 → 2026-09-08 (4000 PRs read, `sort=created&direction=desc` paged
# until the window closed): 6 of 4000 PRs had a base other than `main`
# (0.15%), all six the #14478/#15679 stack. At the window's measured 7008
# pull_request runs per workflow, 0.15% is ~10.5 extra runs per workflow per
# 30 days ≈ 2.5/week, or ~57 job-runs/week across both files at 23 job-runs
# per PR event. Even the pessimistic bound — one uncancelled run per one of
# those PRs' 115 commits — is 1.6% of current pull_request spend. Small, so
# the trigger is widened unconditionally rather than gated on a label or on
# "the base is an open PR's head" (#16482 records the arithmetic).
#
# ⛔ Do NOT mirror this widening onto the `push:` trigger above. `push:` is
# per-branch, not per-PR: unfiltering it would run this workflow on every
# push to every agent branch in the repo, which is a different and far larger
# population than the 6 PRs measured here.
pull_request:
# Merge queue: the queue builds each PR as speculatively merged onto the
# current main and only lands it if this workflow is green on that result —
# the race-free version of the "pull main and re-verify before merging"
# discipline (AGENTS.md multi-agent §7/§10). Every workflow that produces a
# branch-protection-required check MUST carry this trigger, or queue builds
# wait forever on a check that never reports.
merge_group:
# ── THE HOURLY FULL RUN (#16467) ─────────────────────────────────────────
# `push` above now computes the Test Core package set with `--affected`
# against `github.event.before`, so a merge no longer re-tests the whole
# workspace it just tested in the queue. Something still has to run the FULL
# battery on `main`, on a clock, or "main is green" stops being a statement
# about the workspace and becomes a statement about the last diff.
#
# This is that run. Minute 0 rather than an offset minute ON PURPOSE, and
# against the sibling convention in test-nightly-tiers.yml: the reading this
# feeds — `scripts/ci/select-shard-timings-run.mjs` — wants one complete run
# per hour far more than it wants a short queue wait, and a run that starts
# a few minutes late is still the hour's run. What it must NOT do is collide
# with the next hour's, which the concurrency group below decides.
#
# ⛔ This is a NEW TRIGGER KEY, not a widening of `push:`. The ⛔ above still
# binds: `push:` stays filtered to `main`.
schedule:
- cron: '0 * * * *'
# The same full battery, on demand: an operator who has just landed a fix for
# a red hourly run should not have to wait up to an hour to see it answered.
# Treated as `full` by the selection script for the same reason `schedule` is.
workflow_dispatch:
# Superseded runs on the same PR/branch waste runners and delay feedback;
# cancel them. Push runs to main group by commit ref as well, so an in-flight
# main run is cancelled only by a newer main push.
#
# ⭐ `github.event_name` IS LOAD-BEARING IN THIS KEY, and it is the whole
# reason the hourly run above can finish (#16467). On a `schedule` event
# `github.event.pull_request.number` is empty and `github.ref` is
# `refs/heads/main` — BYTE-IDENTICAL to what a push to `main` produces. Without
# the event name in the key the hourly run and the next merge share one group,
# `cancel-in-progress: true` applies, and the next merge kills the hourly run.
# That is not a hypothetical: this card's own measurement is that 36 of the
# last 60 push runs on `main` were already cancelled that way, at a merge
# cadence that would censor most hours.
#
# ⛔ Do not "simplify" this back to two segments. `scripts/ci/scheduled-full-
# run.mjs --check-concurrency` evaluates this expression against a push-shaped
# and a schedule-shaped context and reds when the two groups are equal, so the
# collision cannot come back silently.
#
# `cancel-in-progress` stays `true` for every event including `schedule`: two
# hourly runs overlapping means one of them is over an hour old, and an hour-old
# measurement of `main` is strictly the worse of the two. Test Core's wall clock
# is well inside the cadence (~39 shard-minutes across six shards), so this is
# the exceptional path, not the normal one.
concurrency:
group: ci-${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
filter:
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
pull-requests: read
outputs:
# On merge_group — and on the `schedule` / `workflow_dispatch` full runs
# (#16467) — everything counts as changed: dorny/paths-filter resolves a
# diff only on `pull_request` and `push`, and each of these three events
# is a place a skipped job can never be the right answer (the queue build
# is the last validation before main; the hourly run IS the full battery).
# A skipped step's output is the empty string (falsy), so `|| 'true'`
# supplies their value without touching PR/push behavior.
docs: ${{ steps.changes.outputs.docs || 'true' }}
core: ${{ steps.changes.outputs.core || 'true' }}
console: ${{ steps.changes.outputs.console || 'true' }}
# Gates the `test` job ONLY, ORed with `core` — see the `crosspkg`
# filter below and the `test` job's `if:` (#9829, #10015).
crosspkg: ${{ steps.changes.outputs.crosspkg || 'true' }}
# ── THE FILTER CONTRACT, both halves (#4928) ──────────────────────────
#
# Half 1 is the `|| 'true'` above: when in doubt, RUN EVERYTHING. It
# covers a missing output VALUE.
#
# Half 2 is every downstream `if:` in this file, and it used to be
# missing. GitHub wraps an `if:` that names no status function in an
# IMPLICIT success(), so the original `if: needs.filter.outputs.core ==
# 'true'` had TWO independent paths to "skip" when this job DIED
# (checkout flake, a dorny/paths-filter fault, the 10-minute timeout):
# the implicit success() is false, and the output is no longer a
# trustworthy 'true'. Both roads led to skipping test / dogfood /
# build-core — and skipped counts as SUCCESS in branch protection, so a
# single flake here produced a fully green, zero-test-run, mergeable PR
# with no red signal anywhere.
#
# So every downstream job now spells the contract as "skip only when the
# filter EXPLICITLY said false":
#
# if: ${{ !cancelled() && needs.filter.outputs.<name> != 'false' }}
#
# `!cancelled()` displaces the implicit success() (a genuine run-level
# cancellation still skips — #3668's lifecycle reasoning); `!= 'false'`
# makes the empty string mean "run", exactly like `|| 'true'` does. This
# is deliberately robust to BOTH readings of what a failed job's outputs
# are — null/empty or the `|| 'true'` fallback — because `'' != 'false'`
# and `'true' != 'false'` are both true. The old form was broken under
# both, because the implicit success() dominates either way.
#
# Third occurrence of this GitHub semantic in one audit; the other two
# were release.yml's publish-integrity guard and its docker job (#4900).
steps:
- name: Checkout repository
uses: actions/checkout@v7
# ⭐ SPELLED AS AN ALLOW-LIST, not as `!= 'merge_group'` (#16467). This
# action reads a diff, and the only two events that hand it one it can
# resolve unaided are `pull_request` (base vs head) and `push`
# (`event.before` vs `after`). `merge_group` was the first event with
# neither, and the exclusion was written as its name; `schedule` and
# `workflow_dispatch` are the second and third, and under the old
# spelling they would have RUN the action with no resolvable base.
#
# That failure direction is the dangerous one: every filter output would
# come back `false`, `|| 'true'` supplies nothing (the output exists and
# says `false`), and every downstream `!= 'false'` guard SKIPS — so the
# hourly full run would have been an entirely green, entirely empty run
# of nothing at all. An allow-list makes each new event fail toward
# THE FILTER CONTRACT's half 1 instead: the step is skipped, its outputs
# are the empty string, `|| 'true'` fills them in, and everything runs.
- uses: dorny/paths-filter@v4
id: changes
if: github.event_name == 'pull_request' || github.event_name == 'push'
with:
filters: |
docs:
- 'apps/docs/**'
- 'content/**'
- 'pnpm-lock.yaml'
- '.github/workflows/ci.yml'
core:
- 'packages/**'
- 'examples/**'
- 'apps/!(docs)/**'
- 'package.json'
- 'pnpm-lock.yaml'
- 'tsconfig.json'
- '.github/workflows/ci.yml'
# Build inputs of the vendored Console SPA — see the Console Pin Gate
# job at the bottom of this file. `.objectui-sha` is a ROOT DOTFILE, so
# it matches neither filter above: a pin-only diff skipped `core` and
# `docs` alike, which is how #4288 moved the pin 76 commits with six of
# fourteen checks skipped and nothing anywhere building the new SHA.
# The last two entries are the "a change to the guard runs the guard"
# rule the repo applies to every other filtered gate; they are close to
# free here because an unmoved pin hits the dist cache.
console:
- '.objectui-sha'
- 'scripts/build-console.sh'
- 'scripts/check-console-sha.mjs'
- 'scripts/check-console-injection.mjs'
- 'scripts/console-spec-probes.mjs'
- 'scripts/assert-console-spec-injection.mjs'
- '.github/workflows/ci.yml'
# Test inputs that live OUTSIDE every package (#9829, #10015). Packages
# whose suites read across their own boundary declare that radius in
# scripts/check-cross-package-test-inputs.mjs, and turbo.json mirrors each
# as a `$TURBO_ROOT$/…` input. The step that pulls those packages back into
# the affected set, `--union-into` in the `test` job below, is a step
# INSIDE that job — so when NO filter matches the changed path the job
# never starts, the union never runs, and the declaration buys nothing at
# PR time: the merge queue is the first signal, which is the #7802 failure
# this whole mechanism exists to prevent.
#
# Measured by instantiating every one of turbo.json's `$TURBO_ROOT$`
# declarations to a real tracked file and running it through these filters
# with picomatch — the matcher dorny/paths-filter uses — under BOTH 2.3.1
# (what the action's own lockfile resolves and ncc-bundles) and 4.0.5 (what
# this tree resolves); the two agree on every row. On `0bfbeb70c`, 19 of 64
# declarations were core=false. #9829 closed the nine rooted at `scripts/`;
# the four roots added below are the remaining ten (#10015).
#
# Deliberately a SEPARATE output rather than a widening of `core`, because
# it gates ONE job — the `test` job ORs it with `core`, while Build Core,
# Dogfood and Temporal Conformance stay filtered exactly as they are today.
# That separation is what makes `content/**` affordable at all. Widening
# `core` would put the whole core pipeline on every docs PR; here a
# docs-only diff starts Test Core, whose package set is `turbo ls
# --affected` (empty for such a diff) UNIONED with the packages whose
# declarations matched — so it runs the declaring package, not the
# workspace. Measured through the real `--union-into` on all three
# content-only commits in the window below: `create-objectstack` alone,
# all three times, against a positive control where a
# `packages/lint/src/**` diff unions in @objectstack/spec and
# @objectstack/core.
#
# WIDTH, per entry — each is exactly what some package declared, never a
# root rounded up:
# `scripts/**` and `content/**` are declared verbatim, by
# @objectstack/spec and create-objectstack respectively, so the
# narrower filter does not exist and a hand-kept list of the paths
# inside them is the failure mode #7802 recorded ("a list you must
# remember to update"). Eight further single-file `scripts/`
# declarations and six narrower `content/docs/…` ones (spec, cli,
# dogfood) sit inside the two.
# The last four are a single file, a single subtree, a single file and a
# single file, because that is what the declarations say — NOT `docs/**`,
# `skills/**` or `.github/workflows/**`. Over the last 100 first-parent
# commits of `main` at `0bfbeb70c`, three of those roots would newly have
# started Test Core on 5, 2 and 3 commits; the declared paths on ZERO.
# `skills/objectstack-automation/SKILL.md` joined later (#10178) and was
# NOT in that window's measurement — it is declared by @objectstack/rest,
# one file, and the `skills/**` root it sits in is still declined here.
#
# PRICE, re-measured over that same 100-commit window through this matcher:
# Test Core scheduling goes 80/100 → 83/100. All three additions are
# content-only docs commits, and each runs one package's suite.
#
# This list is a SECOND recognizer of those declarations and is kept in
# step with them BY HAND: nothing fails today when a declaration lands in a
# root no entry here covers. Closing that is #10379.
crosspkg:
- 'scripts/**'
- 'content/**'
- 'docs/audits/2026-07-unknown-key-strictness-ledger.md'
- 'skills/objectstack-formula/**'
- 'skills/objectstack-automation/SKILL.md'
# @objectstack/spec's scripts/export-list.test.ts corpus gate reads
# the whole published catalog (#12201). Verbatim per the declaration;
# it subsumes the two narrower skills entries above, which are left
# as the packages that declared them spelled them.
- 'skills/**'
- '.github/workflows/scaffold-e2e.yml'
- '.claude/skills/spec-property-retirement/SKILL.md'
# @objectstack/lint's production-witness suite reads the repo-root
# SDUI manifest (#12924) — verbatim per the declaration in
# scripts/cross-package-test-inputs.mjs (check:ci-filter-parity
# holds this list to the declarations; the examples/ page files
# the same suite declares are covered by entries the core filter
# already carries).
- 'sdui.manifest.json'
# @objectstack/rest's meta-state-route-doc-spelling test discovers
# its population instead of listing it (#14561): it asks git for
# the authored-prose corpus under content/, docs/ and skills/ and
# judges every file that mentions the route. Verbatim per that
# declaration. `content/**` and `skills/**` above already cover
# two of the three roots; this is the third, and the FIRST entry
# to open the `docs/` root that the WIDTH note above records as
# declined — declined then because no declaration reached past
# `docs/audits/...`, and reached now because one does. The
# narrower audits entry is left as the package that declared it
# spelled it, exactly as the two narrower `skills/` entries were.
- 'docs/**'
test:
# Sharded 6-way BY PACKAGE: a core-touching PR ran the affected suite
# ~11½ min on one 4-vCPU runner — the longest pole in the whole workflow —
# and at 2 shards the slower shard still ran ~10 min, keeping merge-queue
# builds at ~11 min end-to-end (#4859).
# scripts/partition-test-shards.mjs splits the package list into six
# deterministic bins balanced on MEASURED per-package test duration
# (scripts/test-shard-timings.json, generated — see #10472 below) and each
# shard runs its slice through turbo. NOT the dogfood
# job's vitest --shard passthrough, deliberately: that works for dogfood
# because it is ONE package with ~60 files, but applied workspace-wide,
# vitest 4 hard-fails every package with fewer test files than the shard
# count — and `--passWithNoTests` converts the failure into running NOTHING
# on either shard (three packages have exactly one test file today). See
# the script header for the verification.
#
# ── WHY SIX AND NOT THREE (#10227) ────────────────────────────────────
#
# The per-package durations this job's logs print are CONTENDED WALL
# CLOCK, not package cost. `--concurrency=4` on a 4-vCPU runner means up
# to four suites share four cores, so each averages about one core, and
# the contention factor differs per package (measured spread across six
# packages: 0.69x-2.05x). Read as cost, those numbers said the two long
# poles were inherently over #4859's <=7min (420s) line and that no shard
# count could help. Measured like-for-like against the same suites run
# ALONE on an idle 4-core box (#10152 / PR #10258), both clear it:
#
# @objectstack/cli 548.6s contended vs 337.13s alone
# @objectstack/spec 496.4s contended vs 325.31s alone
#
# So the thing over the line is the co-tenancy, not the suites. More
# shards means fewer suites per runner AND four more vCPUs per added
# shard, because every shard gets its own runner.
#
# SIX specifically, and not four or five: sharding is BY PACKAGE, so no
# shard can finish faster than its single heaviest package.
# `@objectstack/spec` is that package — at five shards or fewer the
# partitioner must co-schedule it with others, and at six it very nearly
# fills a bin on its own. Six is therefore the smallest count that
# isolates the heaviest indivisible suite; past six, spec's shard cannot
# improve, only the others can.
#
# ⚠ THE ARGUMENT ABOVE USED TO BE MADE IN TEST-FILE COUNTS (bins
# 415/389/389/389/389/388, spec carrying 415 of ~2360 files). It is now
# made in measured seconds, because #10472 replaced the weight input: the
# file count was a proxy that ran ~2.8x off on @objectstack/cli, and the
# six perfectly count-balanced bins it produced ran 5.0/6.2/6.3/13.6/6.3/
# 9.0 min in run 32428961038. The conclusion (six) survived the re-
# measurement; only the units it is argued in changed. Weights now come
# from scripts/test-shard-timings.json — see partition-test-shards.mjs.
#
# AND WHY NOT MORE THAN SIX, measured on that dataset (#10472 asked for
# 6 -> 8 to be considered). The floor is the heaviest single package, and
# it does not move when shards are added, while the mean falls with every
# shard — so max/mean, which is what the acceptance bound is written in,
# gets WORSE past the point where the floor becomes the max:
#
# 6 shards -> max 650s mean 649s ratio 1.00x
# 7 shards -> max 571s mean 556s ratio 1.03x
# 8 shards -> max 571s mean 487s ratio 1.17x
# 10 shards -> max 571s mean 389s ratio 1.46x
#
# Eight shards would buy ~79s off the critical path for two more runners'
# fixed overhead and a worse balance ratio. The partitioner's self-test
# pins this arithmetic so the next person gets the answer from a failing
# assertion rather than from a CI run.
#
# ⚠ THE COST, stated because it is real: per-shard fixed overhead
# (checkout + pnpm/Turbo cache restore + install, ~60s measured on run
# 32392712980) is paid once PER SHARD, and the merge queue multiplies it
# by the speculative build count. Six shards trade roughly 1.7x the
# billed runner-minutes for a shorter critical path. `--concurrency=4` is
# deliberately UNCHANGED here: narrower shards may well want a lower
# value, but moving both at once makes the next measurement unreadable.
#
# Branch protection requires the bare "Test Core" context, which a matrix
# can never publish again — the test-gate job below carries that name
# (the #3622 lesson; see dogfood-gate).
name: Test Core (${{ matrix.shard }}/6)
needs: filter
# "Skip only when the filter EXPLICITLY said no core paths changed" — see
# THE FILTER CONTRACT on the filter job's outputs (#4928). `scripts` is ORed
# in, never ANDed: two independent reasons to run this job, and it skips only
# when BOTH filters explicitly said false, so the contract's "when in doubt,
# RUN EVERYTHING" posture is unchanged (#9829).
if: ${{ !cancelled() && (needs.filter.outputs.core != 'false' || needs.filter.outputs.crosspkg != 'false') }}
runs-on: ubuntu-latest
# Backstop only — the stall guard on the test steps is the primary
# detector for a #4250-style hang and fires well before this. 30 min is
# ~5× a normal sharded run (~4-6 min), with margin for a cold Turbo cache;
# the old 45 left a hung job "running" for half an hour past any plausible
# healthy finish.
#
# ⚠ TEMPORARY RAISE, 30 -> 45 (#16173). Shard 5/6 is being killed at the
# 30-minute wall — 12+ observations at 30:16-30:21 with every other job in
# every one of those runs green — so the paragraph above describes the
# value this line must RETURN to, not the value it currently carries.
#
# Two reasons, both #16173's:
# 1. Unblock. The only uncensored shard-5 readings are 25:32 and 27:36,
# and shard 6/6 has been seen at 28:56 — a ~1 minute margin at the top
# of the observed green band against a 30:00 wall.
# 2. Un-censor. Every tail observation is a reading of THIS number, not
# of the shard: the true duration is >= 30:16, unbounded above. So the
# shard timings cannot be re-derived from CI history while the wall
# stands here. Raising it produces the durations #16173 needs before
# anyone touches `scripts/test-shard-timings.json`.
#
# ⛔ REVERT CONDITION, explicit: back to `30` once #16173 lands its shard
# rebalance. This value is temporary and carries no other expiry — a raise
# with no revert condition written beside it becomes permanent by
# forgetting. The backstop stays loose only for that window; the stall
# guard named above remains the primary hang detector throughout.
timeout-minutes: 45
permissions:
contents: read
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4, 5, 6]
steps:
- name: Checkout repository
uses: actions/checkout@v7
with:
# Full history so `turbo --affected` can diff against the PR base.
fetch-depth: 0
- name: Setup Node.js
uses: actions/setup-node@v7
with:
node-version: '22'
- name: Setup pnpm
uses: ./.github/actions/setup-pnpm
- name: Get pnpm store directory
shell: bash
run: |
echo "STORE_PATH=$(pnpm store path --silent)" >> $GITHUB_ENV
- name: Setup pnpm cache
uses: actions/cache@v6
with:
path: ${{ env.STORE_PATH }}
key: ${{ runner.os }}-pnpm-store-v3-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-v3-
# Restore-only on PRs: PR-side saves (~5 turbo entries per push) churned
# the repo's 10 GB Actions cache pool and evicted the main-branch seeds —
# observed as sudden cold-cache spikes (Build Core 51s → 4m30s). PRs fall
# back to main's entries via the prefix restore-keys; only main pushes
# save (the "Save Turbo cache" step at the end of the job). Shard-scoped
# key: each shard builds/tests a different half of the workspace (same
# pattern as dogfood). Jobs that used to fall back to this job's
# namespace fall back to Build Core now — neither single shard builds a
# superset anymore.
- name: Restore Turbo cache
uses: actions/cache/restore@v6
with:
path: .turbo/cache
key: ${{ runner.os }}-turbo-${{ github.job }}-${{ matrix.shard }}-${{ github.ref_name }}-${{ github.sha }}
restore-keys: |
${{ runner.os }}-turbo-${{ github.job }}-${{ matrix.shard }}-${{ github.ref_name }}-
${{ runner.os }}-turbo-${{ github.job }}-${{ matrix.shard }}-
${{ runner.os }}-turbo-${{ github.job }}-
- name: Install dependencies
run: pnpm install --frozen-lockfile
# PRs and merge-queue builds: only test packages affected by the diff —
# against the PR base on pull_request, against the group's own
# `base_sha` on merge_group (#16453). spec sits at the root of the
# dependency graph, so spec-touching diffs still run (close to)
# everything — but the many that don't touch spec skip the bulk of the
# 75-package matrix, and at ~120 queue merges a day that is the bulk of
# the queue's test compute. Push to main partitions the FULL package
# list: it gates nobody, and its full run is the ground truth the
# shard-timings refresh reads. (Spec's suite runs here plain /
# uninstrumented; the coverage-instrumented pass lives in the nightly
# coverage-nightly workflow.)
#
# !@objectstack/dogfood: the ~7½-minute dogfood suite is the dedicated
# Dogfood job's whole purpose, and both jobs run under the same `core`
# filter — without the exclusion every core PR executed the suite twice
# in parallel, and it dominated this job's critical path.
#
# `turbo ls` is experimental; the partition script asserts its output
# shape loudly so an upgrade that changes it turns into a red step
# naming the cause, not a silently empty shard. An EMPTY shard file must
# short-circuit the test step below: `turbo run test` with zero --filter
# args runs the entire workspace.
#
# WHERE THE AFFECTED DIFF STARTS (#6195, the #6129 family's third
# consumer). The one thing this base must never be is
# `github.event.pull_request.base.sha`.
#
# The payload's `base.sha` is frozen when the PR is OPENED and does not
# move on `synchronize`. HEAD, meanwhile, is the merge ref
# (`refs/pull/N/merge`) that the checkout above resolves by default on a
# `pull_request` event — no `ref:` is given, so this job stands on a merge
# commit containing everything main has today. Everything main gained
# while the PR sat open therefore lands between the two, and
# `turbo ls --affected` reads it as this PR's own changes: packages only
# SOMEBODY ELSE's merged PR touched get tested on this shard.
#
# The direction is conservative — the frozen base is an ancestor of HEAD,
# so its file set is a strict SUPERSET of this PR's own. Nothing that
# should run is skipped; what degrades is the optimisation itself, and it
# degrades with how long the PR has been open. At ~18 merges a day an
# affected-only shard drifts back toward a full run.
#
# Measured on turbo 2.10.7 against a real merge-ref fixture (base branch
# moved 1 commit touching pkg-b; this PR touched pkg-a only):
# TURBO_SCM_BASE=<frozen base.sha> -> pkg-a, pkg-b
# TURBO_SCM_BASE=merge-base(origin/main,HEAD) -> pkg-a
#
# Four spellings that look like the fix and are not:
# - `base.sha...HEAD` (three dots). Three-dot means
# `merge-base(base.sha, HEAD)..HEAD`, and the frozen sha is ALREADY an
# ancestor of HEAD, so it IS its own merge base and the set does not
# move. Measured: still both packages. (pr-automation.yml records the
# same result for its own diff — same fact, one family.)
# - `HEAD^1`. Correct on a merge ref and silently wrong the day someone
# gives this checkout a `ref:`, where parent^1 becomes the PR's
# previous commit. `merge-base` is right under BOTH checkouts.
# - Dropping the variable and letting turbo default to `main`. Measured
# in a CI-shaped clone: `fetch-depth: 0` populates
# `refs/remotes/origin/*`, NOT local heads, so there is no local `main`
# — and turbo does not error, it silently returns the ENTIRE
# workspace, untouched packages included. Safe, and the whole
# optimisation gone.
# - `TURBO_SCM_BASE=origin/$BASE_REF`, letting turbo resolve the ref.
# Measured correct today, but it rests the shard's package set on
# `turbo ls`'s internal choice of dot-ness — undocumented, and
# `turbo ls` is experimental (see above). Resolving to a commit here
# leaves turbo no choice to make.
# The selection itself lives in scripts/ci/select-shard-packages.sh:
# environment in, `$RUNNER_TEMP/turbo-ls.json` out. This step only
# exports the event's fields, so the same script -- every branch of
# it -- runs locally and under its self-test (#16453). Per event:
# pull_request affected set against merge-base(origin/<base>, HEAD)
# merge_group affected set against the group's `base_sha`, unioned
# with the same cross-package scans. An EMPTY set is a
# legitimate docs-only group: it reaches the "No packages
# on this shard" exit below, every shard still attests,
# and Test Core is an honest green. It is NOT the #10057
# case, which is pull_request-only (the script says why).
# push affected set against `github.event.before`, the commit
# `main` was on before this merge landed (#16467). A zero
# sha -- a first push, or a force-push that rewrote
# history -- falls back to the FULL list, loudly.
# schedule the FULL list: the hourly run IS the full battery, and
# workflow_ the on-demand rerun of it. Neither carries a diff base,
# dispatch and neither should: they are the run that says whether
# the whole workspace is green on `main`.
- name: Compute this shard's package set
env:
OS_SHARD_EVENT_NAME: ${{ github.event_name }}
OS_SHARD_PR_BASE_REF: ${{ github.event.pull_request.base.ref }}
OS_SHARD_PR_PINNED_BASE_SHA: ${{ github.event.pull_request.base.sha }}
OS_SHARD_MERGE_GROUP_BASE_SHA: ${{ github.event.merge_group.base_sha }}
OS_SHARD_PUSH_BEFORE_SHA: ${{ github.event.before }}
run: |
bash scripts/ci/select-shard-packages.sh
node scripts/partition-test-shards.mjs "$RUNNER_TEMP/turbo-ls.json" \
--shard ${{ matrix.shard }}/6 --exclude @objectstack/dogfood \
> "$RUNNER_TEMP/shard-packages.txt"
echo 'Items on this shard (a package name, or a package plus a k/n file-level slice):'
cat "$RUNNER_TEMP/shard-packages.txt"
# ⛔ A FILE-LEVEL SLICE BUILDS ITS DEPENDENCY CLOSURE HERE, IN A RUN THAT
# CARRIES NO PASSTHROUGH, so that the sharded run in the next step can be
# `--only` (#16395).
#
# Turbo folds a run-level passthrough into the hash of EVERY task in the
# run, not only the task that receives it -- and `-- "--shard=k/n"` is the
# whole reason a slice gets its own invocation at all (the next step's
# comment says why it cannot ride the shared run). Measured on turbo
# 2.10.10, `--filter=@objectstack/cli`, `turbo run test ... --dry=json`
# (60 tasks: 59 `build` + 1 `test`):
#
# plain vs plain 60 identical, 0 changed <- control, fires
# plain vs -- --shard=1/2 0 identical, 60 changed
# --shard=1/2 vs 2/2 0 identical, 60 changed
#
# So the sliced leg could hit NEITHER the main-seeded Turbo cache restored
# above NOR the builds the shared leg ran seconds earlier in the SAME job:
# it re-executed the closure every run, and shards 5/6 and 6/6 -- the two
# that carry a slice -- paid that closure twice per job. Live reading,
# `Test Core (5/6)` of run 34193080219 (a `packages/spec` PR, so the
# affected set reaches cli): the slice leg reported
# `Cached: 2 cached, 58 total` / `Time: 9m18.941s` while
# `@objectstack/cli:test` itself measured `Duration 187.49s` -- six of those
# nine minutes were the duplicate rebuild, on a job that was then standing
# against a 30-minute wall (#16395's measurement).
#
# `turbo run build --filter=$PKG` is that closure and nothing more:
# measured 59 build tasks, all 59 hash-IDENTICAL to the ones in the
# passthrough-free test plan (0 differing, 0 extra, and 0 missing against
# the test's own `^build` closure), so they REPLAY rather than re-execute.
# `--filter=...^$PKG` was measured too and schedules 12 packages this
# closure does not need. Locally, back-to-back invocations of this exact
# command measured `57 cached, 57 total` / `Time: 153ms >>> FULL TURBO` on
# the second, against `5 cached, 57 total` / `3m45.918s` on the first.
#
# ⚠ THIS IS ITS OWN STEP, not a second guarded run inside the step below,
# because a guarded SITE is the triple (file, job, step) --
# `measure-stall-guard-headroom` REFUSES to report a verdict when two
# guarded runs share one, and refusing is right: the two would be judged
# against the worst reading of their union. `pnpm check:stall-guard-budget`
# and `pnpm check:stall-guard-headroom` both read this step, so it keeps
# its own `--stall-minutes` and its own headroom row.
#
# A shard with no slice runs zero iterations here; every shard still
# reaches the step, so its name is a stable site for those two gates.
- name: Build the sliced package's dependency closure
env:
NODE_OPTIONS: --report-on-signal --report-signal=SIGUSR2 --report-directory=${{ runner.temp }}/stall-reports
run: |
if [ ! -s "$RUNNER_TEMP/shard-packages.txt" ]; then
echo "No packages on this shard — nothing to build."
exit 0
fi
mkdir -p "$RUNNER_TEMP/stall-reports"
while read -r PKG SLICE; do
[ -n "$PKG" ] || continue
[ -n "$SLICE" ] || continue
echo "Slice $PKG=$SLICE — building its dependency closure with no passthrough."
LOG="$RUNNER_TEMP/test-core-slice-build-$(printf '%s' "$PKG" | tr -c 'A-Za-z0-9' '-').log"
node scripts/run-with-stall-guard.mjs --log "$LOG" --stall-minutes 10 \
--report-dir "$RUNNER_TEMP/stall-reports" -- \
pnpm turbo run build "--filter=$PKG" --concurrency=4 --log-order=stream
done < "$RUNNER_TEMP/shard-packages.txt"
# --concurrency=4: turbo's default (10) oversubscribes the 4-vCPU
# hosted runner; matching the core count bounds peak memory and the
# job is CPU-bound anyway.
# run-with-stall-guard replaces the old `… 2>&1 | tee $log` +
# `set -o pipefail` idiom: the guard tees combined output to the log
# itself and propagates the suite's real exit status, so there is no
# pipe whose status tee could mask (do not reintroduce `| tee`). Its
# actual job is #4250: a run whose output freezes mid-suite while the
# job sits in_progress. Silence past --stall-minutes is declared a
# stall — a labeled red naming the last output line — instead of a
# 20-minute wait for a human (or the job timeout) to notice. 10 min is
# ~5× the longest healthy quiet gap.
#
# NODE_OPTIONS arms every node process (vitest workers included) to dump
# a diagnostic report on SIGUSR2; on a stall the guard signals the frozen
# process group and digests the reports into the log — exact JS stack for
# a process whose event loop is alive, and a named "no report = blocked
# loop" verdict for one that is sync-spinning. The next #4250 occurrence
# identifies its own culprit instead of costing a diagnosis.
#
# --log-order=stream is MANDATORY under the guard, not a preference. The
# guard measures output FLUSHES; turbo's default in CI is grouped log
# order, which flushes a task's output only when the task ENDS — so a
# shard whose tail is one task longer than --stall-minutes was killed BY
# CONSTRUCTION, healthy or not (measured: a healthy cli:test killed at
# exit 75 with 173/173 files passing in the flush the kill forced). The
# guard now refuses to wrap a turbo run without this flag. Cost is
# interleaved logs; the completeness guard below reads the per-line
# `<pkg>:test:` prefixes stream order emits (a pinned parseSummaries
# branch), so attribution survives.
- name: Run this shard's tests
env:
NODE_OPTIONS: --report-on-signal --report-signal=SIGUSR2 --report-directory=${{ runner.temp }}/stall-reports
# The `e2e` and `live` filename tiers run NIGHTLY on main
# (test-nightly-tiers.yml), not here: `queue` is the per-PR and
# merge-queue setting, read once in scripts/nightly-tiers.mjs. Spelled
# explicitly even though unset reads the same, so the setting this
# required check verifies is written where the check runs.
# turbo.json hashes it in the `test` task's `env`, which is what lets
# it reach vitest under strict env mode at all.
OS_TEST_TIERS: queue
run: |
if [ ! -s "$RUNNER_TEMP/shard-packages.txt" ]; then
echo "No packages on this shard — nothing to test."
exit 0
fi
# Bound vitest's INNER worker pool (#11958). A no-op on a runner with
# <= 5 cores (it only ever LOWERS vitest's own `cores - 1` default);
# it exists so a larger runner cannot multiply turbo's outer
# --concurrency by a host-sized inner pool. Empty on failure, which is
# vitest's own "use the default" signal. Needs turbo.json's
# globalPassThroughEnv entry or turbo strips it — see the script header.
export VITEST_MAX_WORKERS="$(node scripts/vitest-worker-cap.mjs)"
mkdir -p "$RUNNER_TEMP/stall-reports"
# Split the shard's ITEMS into the whole packages, which share one
# turbo run as they always have, and the file-level slices, which
# cannot: `--shard=k/n` is passed through to vitest by turbo as a
# RUN-level argument, so it would reach every package in the run —
# and on any package with fewer test files than n that is a hard
# vitest failure (or, with --passWithNoTests, silently no tests at
# all). A slice therefore gets its own invocation, filtered to the one
# package the partitioner sliced.
FILTERS=""
SLICES=""
while read -r PKG SLICE; do
[ -n "$PKG" ] || continue
if [ -n "$SLICE" ]; then
SLICES="$SLICES $PKG=$SLICE"
else
FILTERS="$FILTERS --filter=$PKG"
fi
done < "$RUNNER_TEMP/shard-packages.txt"
# Each leg tees to its OWN log — run-with-stall-guard opens the log
# with 'w', so a second leg pointed at one path would truncate the
# first leg's output and the completeness guard below would grade half
# a shard. They are concatenated afterwards, and that concatenation
# happens even when a leg failed, because a red suite is exactly when
# the completeness guard earns its keep.
#
# ⛔ A failing leg STOPS the remaining ones, the same way turbo stops
# scheduling on the first failure inside one run. Carrying on would add
# a second full suite to a job that is already red and already inside a
# 30-minute wall — turning an informative red into a killed job with no
# attestation at all, which is the #16173 failure mode itself.
#
# `test test:repo` (#16466): six heavy packages split their suite into
# two turbo tasks -- `test` (package-local inputs) and `test:repo`
# (the repo-scanning tests, hashed on the wide `$TURBO_ROOT$` inputs
# that used to sit on `test`). Naming both runs a split package whole
# and is a no-op for every package without a `test:repo` script. The
# per-shard Turbo cache restored above (main-seeded) is what carries
# an unmoved `PKG#test` across a change outside the package; the
# completeness guard below reads both tasks' summaries. The cli slice
# leg stays `test`-only: cli is not split.
STATUS=0
LOGS=""
for LEG in __whole__ $SLICES; do
if [ "$LEG" = __whole__ ]; then
[ -n "$FILTERS" ] || continue
LOG="$RUNNER_TEMP/test-core-packages.log"
set -- pnpm turbo run test test:repo $FILTERS --concurrency=4 --summarize --log-order=stream
else
PKG="${LEG%%=*}"
SLICE="${LEG#*=}"
LOG="$RUNNER_TEMP/test-core-slice-$(printf '%s' "$PKG" | tr -c 'A-Za-z0-9' '-').log"
# `--only` (#16395): the step above already built this slice's
# dependency closure in a passthrough-free run, so this run must
# schedule the ONE task the passthrough is for. Without it turbo
# re-hashes the whole `^build` closure under `--shard=k/n` and
# rebuilds it -- that comment carries the measurement. ⚠ The build
# step is load-bearing for this flag: a sliced package whose build
# never ran fails LOUDLY here (its imports resolve to a missing
# dist), never as a silent green.
set -- pnpm turbo run test "--filter=$PKG" --only --concurrency=4 --summarize --log-order=stream -- "--shard=$SLICE"
fi
LOGS="$LOGS $LOG"
node scripts/run-with-stall-guard.mjs --log "$LOG" --stall-minutes 10 \
--report-dir "$RUNNER_TEMP/stall-reports" -- "$@" || { STATUS=$?; break; }
done
# Only over logs that EXIST, and via an explicit `if` rather than a
# `&&` chain: a leg whose guard never got far enough to open its log
# would otherwise make `cat` non-zero, and under `set -e` that replaces
# the SUITE's exit status with cat's — the step would report the wrong
# reason for its own red.
: > "$RUNNER_TEMP/test-core.log"
for LOG in $LOGS; do
if [ -f "$LOG" ]; then
cat "$LOG" >> "$RUNNER_TEMP/test-core.log"
fi
done
exit $STATUS
# --summarize above costs nothing at runtime and writes
# `.turbo/runs/<id>.json`: one per-task record with the execution window
# turbo actually observed. That file IS the balancing input for the next
# shard split (scripts/test-shard-timings.json is generated from it), and
# publishing it here is what keeps the dataset re-measurable instead of
# hand-frozen — refreshing it becomes "download these six artifacts from
# any green run and re-run the generator", with no special measurement
# run and no CI time spent measuring.
#
# `if: always()` so a shard that went red still publishes what it timed;
# the generator refuses failed and cache-replayed tasks on its own side,
# so a partial upload contributes nothing rather than something wrong.
#
# ⛔ This step must stay ABOVE the attestation pair at the bottom of the
# job. check-shard-attestation.mjs requires `--emit` + upload-artifact to
# be the job's LAST TWO steps, because anything after them can fail the
# job while the credential already counts as a pass (#6082). Adding an
# upload here is fine; moving it below them is not.
#
# continue-on-error: this upload is diagnostics-only (`if: always()` and
# `if-no-files-found: ignore` already say so), but being unguarded still
# put it on the shard's pass/fail path: a failed *finalize* aborts the
# job before the attestation pair below ever runs, so a platform-side
# fault here evicted a fully green shard from the merge queue (#14469 —
# FinalizeArtifact 403 on a 313-byte upload, content already stored).
# The attestation pair is untouched by this flag and remains the
# fail-closed credential exactly as before — it still runs, still has
# no `continue-on-error`, and still must succeed (#6082 unweakened).
# ⛔ Do not add continue-on-error, a retry, or an `if:` to the
# attestation pair below to "match" this step — that pair is the
# credential, not diagnostics, and check:shard-attestation pins its
# unguarded shape statically.
- name: Publish this shard's turbo run summary
if: always()
continue-on-error: true
uses: actions/upload-artifact@v7
with:
name: test-core-run-summary-${{ matrix.shard }}-of-6
path: .turbo/runs/
if-no-files-found: ignore
retention-days: 1
# ⛔ THE DRIFT STEP IS DELIBERATELY NOT WIRED HERE YET (#16173).
#
# partition-test-shards.mjs carries a fully-tested `--check-drift` mode —
# it reads the summary uploaded above back and reds when a shard's MEASURED
# test total outruns its PREDICTED one past MAX_MEASURED_OVER_PREDICTED. The
# code, its self-tests and its ablation are all on this branch; only this
# invocation waits, and the wait is a SEQUENCING decision, not an oversight.
#
# Why: scripts/test-shard-timings.json is still stale for @objectstack/cli
# (458.15s recorded, 1231.52s measured), so wiring the step today would red
# the shard carrying a CLI slice on every single PR — a true reading, but one
# that blocks everything until the dataset is refreshed. The file-level split
# above already removes the urgent hazard on its own, taking the worst shard
# from ~1445s (80% of this job's 30-minute wall) to ~1059s (59%) with the
# dataset untouched, because the CLI is halved across two runners instead of
# falling on one.
#
# ⇒ The refresh and this step land TOGETHER in the follow-up, in that order.
# The refresh recipe is in PR #16220's body. When it lands, restore a step
# here that runs `--check-drift` over `.turbo/runs/*.json` with
# `--label "Test Core (${{ matrix.shard }}/6)"`, with NO `if:` and NO
# `continue-on-error` (the point is the red), placed ABOVE the attestation
# pair for the #6082 reason documented on the upload above — so a drift red
# also withholds the attestation, which is the fail-closed direction.
# Runs even when the suite failed — that is when it earns its keep. It
# answers TWO questions about a red suite, and needs both to be able to
# say anything at all about a green one.
#
# 1. Was every test vitest COUNTED actually run (#3812)? A worker dying
# at the process level leaves a summary that still leads with
# "passed" while falling short of its own declared count — a red
# that READS like a pass.
# 2. Did every package scheduled on this shard report AT ALL (#10032)?
# Question 1 is answered from summary lines PRESENT in the log, so a
# package that printed nothing contributes no row: neither counted
# nor missed, simply invisible. That is why this step is handed the
# shard's scheduled package list (and the `turbo ls` document it was
# sharded from, which resolves each name to a directory) instead of
# grading the log against itself.
#
# ⛔ THE OLD NOTE HERE WAS WRONG AND COST A REAL TRIAGE. It read "a red
# suite plus a GREEN completeness check means real test failures". That
# inference does not hold for case 2, and #10032 is the bill: Test Core
# (2/3) failed naming @objectstack/example-showcase#test while this guard
# printed OK, because the failing package was the one that printed
# nothing — its complete 5083-line job log mentions that package twice,
# both times in the error summary, with no vitest output anywhere.
#
# ⛔ AND THE REPLACEMENT WAS STILL BLIND, FOR A SECOND REASON — measured
# on run 32391924279 (`Test Core (3/3)`, job 96499888763, complete 31839-
# line log). Under grouped log order turbo folds each task into a
# collapsible `::group::` EXCEPT the one it surfaces as the run's failure,
# which gets a bare `<pkg>:test` header instead. The guard read groups and
# prefixes but not that third shape, so the FAILING package — the one it
# exists to grade — was the one package whose summary it could not
# attribute, and its "refuse to guess" backstop then withheld judgement on
# every candidate and exited 0. It printed `OK (24 of 29 …)` under a note
# naming @objectstack/example-showcase as ungraded: the same green-that-
# reads-like-a-clearance, one turn of the screw further out.
# ⇒ The guard now knows all three header shapes, a package named in
# turbo's `Failed:` roster is judged even when the log holds strays it
# cannot place, and an ungraded package downgrades the verdict word from
# OK to PARTIAL and raises a ::warning:: annotation. Only with all three
# does green cover both questions — which is what makes the dichotomy
# above true. ⚠ A red HERE can never turn a green shard red: it needs
# turbo to have named a failed task, and that already failed the run.
#
# ⚠ The guard does NOT charge a silent package to this shard when turbo
# simply never reached it — turbo stops scheduling on the first failure,
# so an ordinary red suite leaves later packages unrun. Those are printed
# as notes. See the script header for the two exemptions and the two
# rules that decide red.
#
# ⚠ NOT diagnosed, by either pass: the original zero-output event itself.
# It has never been reproduced. The 2026-08-20 recurrence was NOT one —
# that failing suite streamed ~940 lines and a complete summary
# (`Test Files 21 passed (21)` / `Tests 342 passed (342)`), and failed on
# a vitest teardown error, not on silence. Read a "zero output" claim as
# a hypothesis until someone has grepped the WHOLE log for it.
- name: Test completeness guard
if: always()
run: |
if [ ! -f "$RUNNER_TEMP/test-core.log" ]; then
echo "No test log — the test step did not get far enough to produce one."
exit 0
fi
node scripts/check-test-completeness.mjs "$RUNNER_TEMP/test-core.log" \
--scheduled "$RUNNER_TEMP/shard-packages.txt" \
--package-list "$RUNNER_TEMP/turbo-ls.json"
# ── Per-file test timings, captured for the aggregator (#16454) ───────
#
# Cutting tests is decided per test file BY COST, so the cost has to be
# visible first. This shard reads the timings it already printed and
# hands them up; `Test Core` merges the six and publishes the table.
#
# WHERE THE NUMBERS COME FROM, and why no second test run is needed:
# per FILE the vitest module lines already in `test-core.log`.
# Measured before this was written, because the default
# reporter appends a file's duration only when the module
# HAS one: on run 34317273493 every file line carried its
# milliseconds, including files far under the 300ms
# slowTestThreshold (which this repo never sets — it picks
# the colour, not the presence). So the stream is enough
# and no machine-readable reporter is added to the shard's
# 30-minute wall. The script's header carries the
# measurement and its control.
# per PACKAGE `.turbo/runs/*.json`, the same `--summarize` output the
# step above already uploads — turbo's own execution
# windows, the unit scripts/test-shard-timings.json was
# measured in, so the drift column compares like with
# like. Cache REPLAYS and FAILED tasks are refused there
# rather than averaged in as near-zero measurements.
#
# ⛔ REPORT ONLY — `if: always()` + `continue-on-error: true`, both
# load-bearing and for different reasons. `always()` so a red shard still
# contributes what it timed (a slow suite is most interesting when it has
# just failed), and `continue-on-error` because an unguarded step here
# sits on the shard's PASS/FAIL path: it would abort the job before the
# attestation pair below ever ran, evicting a green shard from the merge
# queue on a diagnostics fault. That is #14469, measured on the upload
# step above. This card's ruling is report only, no red.
#
# ⛔ These two steps must stay ABOVE the attestation pair at the bottom of
# the job, exactly like the run-summary upload above:
# check-shard-attestation.mjs requires `--emit` + upload-artifact to be
# the job's LAST TWO steps and pins that statically (#6082). Adding steps
# here is fine; moving them below that pair is not — and ⛔ that pair does
# not acquire an `if:` or a `continue-on-error:` to "match" these.
#
# The `--self-test` runs here because this is where CI names the script,
# which is what check:self-test-wired requires. ⚠ Under
# `continue-on-error` its verdict is VISIBLE but not enforcing: inside
# this workflow, no step this card may add is permitted to fail anything.
# Giving that self-test teeth means a `check:` step in lint.yml, which is
# deliberately NOT taken here.
- name: Capture this shard's test timings
if: always()
continue-on-error: true
run: |
if [ ! -f "$RUNNER_TEMP/test-core.log" ]; then
echo "No test log — the test step did not get far enough to produce one."
exit 0
fi
node scripts/report-test-timings.mjs --self-test
node scripts/report-test-timings.mjs --capture \
--log "$RUNNER_TEMP/test-core.log" \
--summaries .turbo/runs \
--shard "${{ matrix.shard }}/6" \
--out "$RUNNER_TEMP/test-timings/shard-${{ matrix.shard }}.json"
# 1 day, matching the run-summary upload above: this is an intermediate
# the aggregator consumes minutes later in the SAME run. The 14-day
# retention this card asks for is on the MERGED table, published by
# `Test Core`.
- name: Publish this shard's test timings
if: always()
continue-on-error: true
uses: actions/upload-artifact@v7
with:
name: test-core-timings-${{ matrix.shard }}-of-6
path: ${{ runner.temp }}/test-timings/
if-no-files-found: ignore
retention-days: 1
# A stall's full diagnostic reports (JS stacks, libuv handles, heap
# summary per process) outlive the in-log digest — keep them so a #4250
# occurrence can be dissected offline. Free when nothing stalled: the
# directory is empty and if-no-files-found skips the upload. Shard-scoped
# name so the two matrix jobs don't collide.
- name: Upload stall diagnostic reports
if: failure()
uses: actions/upload-artifact@v7
with:
name: stall-reports-test-core-${{ matrix.shard }}
path: ${{ runner.temp }}/stall-reports/
if-no-files-found: ignore
retention-days: 14
# Seed the shared Turbo cache from main only (see the restore step
# above). always(): keep the seed fresh even when a test fails, matching
# the old actions/cache post-step behavior. Shard-scoped key so the two
# matrix jobs don't collide.
- name: Save Turbo cache (main only)
if: always() && github.event_name == 'push'
uses: actions/cache/save@v6
with:
path: .turbo/cache
key: ${{ runner.os }}-turbo-${{ github.job }}-${{ matrix.shard }}-${{ github.ref_name }}-${{ github.sha }}
# ── This shard's positive attestation (#6082) ─────────────────────────
# The credential test-gate counts: "shard N of 6 ran, and every step of it
# passed". These two steps are LAST and carry NO `if:` — that is what
# makes the statement true, because an unguarded step runs only when every
# preceding step of the job succeeded, and nothing after them can fail the
# job while the credential already counts as a pass.
# check:shard-attestation enforces both properties statically, so a step
# appended below here, or an `if:` added to either, is a red lint job
# rather than a silently weakened credential.
- name: Attest this shard ran and passed