[DRAFT]Dispatch nightly-stress-test.yml per-branch (master/3006.x/3008.x), each against its own --ref - #70087
Draft
charzl wants to merge 11 commits into
Draft
[DRAFT]Dispatch nightly-stress-test.yml per-branch (master/3006.x/3008.x), each against its own --ref#70087charzl wants to merge 11 commits into
charzl wants to merge 11 commits into
Conversation
schedule/workflow_dispatch triggers only ever run a workflow as it exists on the default branch. 3008.x and 3006.x each already carry their own copy of this file with the same schedule block, but it has never fired for either (confirmed via the Actions run history: every event:schedule run is on master). Matrix the existing job over branch: [master, 3008.x, 3006.x] with an explicit `ref: matrix.branch` checkout, so master's live schedule drives all three -- each running its own checked-out tests/monitoring/ code. Branch-qualify the Docker layer cache key, the uploaded artifact name, and the stress-snapshots publish directory, since matrixed jobs share one github.run_id/github.sha and would otherwise collide. 3007.x is left out: its tests/monitoring/ predates render_panels.py, so the Render Dashboard Panels / Publish panels / Panel Summary steps would fail against it as-is.
Scoped to this branch only -- lets a manual workflow_dispatch actually execute (to verify the checkout-against-fork failure from earlier is fixed by testing directly against this repo's own 3006.x/3008.x) without touching the repo-wide variable, which would also affect master's real schedule and anyone else relying on it. Must be reverted before merge.
VCOPS-100029: today there's no way to run this stress test against a specific PR/branch's diff -- workflow_dispatch only exposes `duration`, and the matrix's checkout ref was hardcoded to [master, 3008.x, 3006.x] regardless of which ref you dispatched against. Compute the matrix branch list from the trigger type instead of a fixed array: schedule (nightly cron) still expands to the same three-branch set unconditionally; workflow_dispatch expands to a single-element array built from the new `branch` input (default "master", but any branch/PR ref the caller types in). event_name=='schedule' short- circuits before `inputs.branch` is evaluated, since schedule events have no `inputs` context.
charzl
changed the title
Drops the matrix/fromJSON trick entirely. nightly-stress-test.yml goes back to a single job with one branch input (default master) -- whoever dispatches it picks exactly one branch, full stop, no schedule trigger of its own anymore. Nightly coverage of master/3008.x/3006.x now comes from a new sibling workflow, nightly-stress-test-dispatch.yml, which holds the only live schedule (registered on master, per GitHub's default-branch-only rule) and fires nightly-stress-test.yml three separate times via workflow_dispatch -- once per branch, each its own independent run rather than a matrix leg sharing one. Always dispatches against --ref master so every run loads the same simple job definition, threading the actual branch through as an input instead. Since each dispatch is now its own run, matrix-driven collisions (shared github.run_id/github.sha across legs) no longer apply -- dropped the RUN_DIR/artifact-name branch-qualification comments that were about avoiding those, kept the qualification itself for readability. Still need inputs.branch (not github.sha alone) in the Docker cache key: every dispatch shares master's github.sha (since dispatch ref is always master), but checks out different code per inputs.branch, so a sha-only key would serve one branch's cached layers to another. 3007.x still deliberately excluded from the nightly set (predates render_panels.py).
The dispatcher (schedule-triggered, fires the nightly set) now owns the nightly-stress-test.yml name -- that's the one people actually mean by "the nightly stress test". The single-run building block (branch + duration inputs, one job) moves to run-stress-test.yml, invoked both by the dispatcher and by anyone manually testing a specific branch/PR. Also trimmed the `branch` input description down to just "Branch", and dropped a comment on the stress-snapshots RUN_DIR that was explaining something not worth explaining.
It gates whether the *nightly* firing happens, not whether a stress test can run at all -- it belongs on nightly-stress-test.yml's dispatch job, not on run-stress-test.yml's, so that manually running run-stress-test.yml against a PR branch (VCOPS-100029) still works regardless of this repo-level variable. Also drops the TEMPORARY if:true bypass from run-stress-test.yml now that the real check has moved off of it entirely.
Mirrors run-nightly.yml's trigger-branch-nightly-builds job: use --ref matrix.branch (via strategy.matrix) instead of always --ref master, so each branch's own copy of run-stress-test.yml governs its run instead of borrowing master's. This means 3008.x and 3006.x each need their own run-stress-test.yml landed (separate PRs against those base branches, coming next) -- master alone isn't enough anymore.
Closed
2 tasks
Per Option A (single source of truth on master): rather than 3008.x keeping its own divergent copy of this workflow just to expose these two inputs, fold them into master's copy as universal, opt-in inputs available to every branch dispatched against. worker_threads defaults to empty (not 3008.x's original '5') -- master's and 3006.x's tests/monitoring/master.conf both currently ship worker_threads: 10, and a hardcoded '5' default would silently nudge their runs into an unrequested config change + restart the first time someone runs this without setting the input. Only a non-empty explicit value touches master.conf now.
Daniel Wozniak (core maintainer) confirmed: workflow_dispatch --ref X runs the copy of the target workflow file that exists ON BRANCH X, not master's copy checked out against X. The previous design (--ref master for every dispatch, reusing master's single-branch-input copy of run-stress-test.yml, with inputs.branch only steering the actions/checkout step inside it) therefore never actually picked up a change made to run-stress-test.yml on a non-master branch -- editing 3006.x's copy of that workflow would silently have zero effect on 3006.x's nightly run. Now each branch in the loop dispatches with --ref matching itself, so that branch's own copy of run-stress-test.yml governs its own nightly run. This file is meant to be byte-identical across master/3006.x/3007.x/ 3008.x once each has it -- porting to a not-yet-covered branch is a straight copy of the whole file, not a re-edit of the branch list. Only master's copy is ever actually fired by the `schedule:` cron (GitHub only honors schedule off the default branch); the branch list still lives here (not only reasoned about on master) so a manual workflow_dispatch of this file on another branch behaves the same as the real nightly run would, and so porting requires no re-derivation. 3008.x is in the branch list already even though it doesn't have its own run-stress-test.yml yet -- that dispatch will fail until it's ported (companion PR), and `|| echo ...` keeps that one failure from blocking master/3006.x in the same run. 3007.x stays out of the list entirely (not commented out) -- its tests/monitoring/ predates render_panels.py, so the Render Dashboard Panels / Publish panels / Panel Summary steps would fail against it even once ported. Also fixes a stale comment on run-stress-test.yml's Docker cache key that described the old always-master dispatch behavior.
charzl
changed the title
2 tasks
charzl
added a commit
that referenced
this pull request
Aug 19, 2026
Companion to run-stress-test.yml added earlier in this branch. Per the design settled in #70087: only master's copy of this dispatcher is ever actually fired by the schedule: cron (GitHub only honors schedule: off the default branch), but every branch in the nightly set carries its own byte-identical copy so that: - a manual workflow_dispatch of this file on this branch behaves the same as the real nightly run would, and - porting to a not-yet-covered branch (3007.x once it's backported) is a straight copy of the whole file, not a re-derivation of the branch list or the --ref-per-branch dispatch logic. Without this file present here, master's dispatcher's `gh workflow run run-stress-test.yml --ref 3006.x` would 404 -- workflow_dispatch --ref X requires the target workflow to exist on branch X, but the dispatch call itself (this file, on master) is a different workflow from the one it's calling out to.
charzl
changed the title
charzl
added a commit
that referenced
this pull request
Aug 21, 2026
Companion to run-stress-test.yml added earlier in this branch. Per the design settled in #70087: only master's copy of this dispatcher is ever actually fired by the schedule: cron (GitHub only honors schedule: off the default branch), but every branch in the nightly set carries its own byte-identical copy so that: - a manual workflow_dispatch of this file on this branch behaves the same as the real nightly run would, and - porting to a not-yet-covered branch (3007.x once it's backported) is a straight copy of the whole file, not a re-derivation of the branch list or the --ref-per-branch dispatch logic. Without this file present here, master's dispatcher's `gh workflow run run-stress-test.yml --ref 3006.x` would 404 -- workflow_dispatch --ref X requires the target workflow to exist on branch X, but the dispatch call itself (this file, on master) is a different workflow from the one it's calling out to.
charzl
marked this pull request as draft
August 21, 2026 17:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
schedule:/workflow_dispatch:triggers only ever run a workflow as it exists on the default branch, so anightly-stress-test.ymlliving only onmasteris the only thing that can actually fire the nightly cron. This dispatchesrun-stress-test.ymlonce per branch in the nightly set (master,3006.x,3008.x) viaworkflow_dispatch.Superseded an earlier "always
--ref master" design. Daniel Wozniak (core maintainer) confirmedworkflow_dispatch --ref Xruns the copy of the target workflow file that exists on branch X, not master's copy checked out against X. So dispatching everything with--ref master(withinputs.branchonly steering theactions/checkoutstep inside that file) meant a change torun-stress-test.ymlmade on e.g.3006.xwould never take effect on3006.x's own nightly run -- master's copy would silently govern every branch's job definition instead. Fixed: each branch now dispatches with--refmatching itself, so that branch's own copy ofrun-stress-test.ymlgoverns its own run.This file is meant to be byte-identical across
master/3006.x/3007.x/3008.xonce each has it -- porting to a not-yet-covered branch is a straight copy of the whole file, not a re-edit of the branch list. Only master's copy is ever actually fired by the cron (GitHub only honorsschedule:off the default branch); the branch list still lives on every copy (not reasoned about only on master) so a manualworkflow_dispatchof this file on another branch behaves the same as the real nightly run would, and porting requires no re-derivation.3008.xis in the branch list already even though it doesn't have its ownrun-stress-test.ymlyet -- that dispatch will fail (workflow not found on that branch) until it's ported;|| echo ...keeps that one failure from blockingmaster/3006.xin the same run.3007.xstays out of the branch list entirely (not commented out) -- itstests/monitoring/predatesrender_panels.py, so the Render Dashboard Panels / Publish panels / Panel Summary steps would fail against it even once ported.3006.x's ownrun-stress-test.ymlto follow.Note for maintainers:
SKIP_NIGHTLY_STRESS_TESTis currentlytrueat the repo level, so this job is skipped rather than actually running right now. That's unrelated to this change -- flagging it since merging this alone won't make the nightly cron start firing until that variable is flipped.Test plan
pre-commithooks pass (workflow lint/template checks included).SKIP_NIGHTLY_STRESS_TESTallows it to run), confirmmaster/3006.xfire on the0 2 * * *cron, each using its own branch's copy ofrun-stress-test.yml; confirm3008.x's dispatch fails gracefully without blocking the others until it's ported.