Repository navigation
CI COST P0 — eliminate duplicate PR waves and repeated Windows builds (P10.9K) #542
Description
Incident: P10.9K CI fan-out wastes capacity and delays real tests
No workflow/code changes are being made by this issue. Issue creation does not trigger PR build workflows.
Measured evidence (12 Oct 2026)
- Exact P10.9K consumer source:
069890df0a007d196ba0a9498c2c5972f8ab5c5b, ARSAS PR P10.9K: vendor-neutral control safety, typed GUI and pinned association inventory engine #541, enginea43b9b4864a9565c02f544d4135fe157c2fac577(engine PR Release ARSAS 1.6.28 #168). - GitHub Actions API for that exact source returns 30 workflow runs, all pull_request, in two waves: 14 runs around 09:55:22 UTC and 16 runs around 10:02:50 UTC. Same final head SHA, repeated families and Windows jobs.
- Exact canonical Build ARSAS run #38233003891 was SUCCESS before the later canonical run #38233823373. This proves repeated expensive work, not just a high count of unique protection checks.
.github/workflows/build.ymlhas unconditionalpull_request:and Windows build/test/package; the 2nd run also succeeded and generated P10.9K portable artifact..github/workflows/smart-discovery-capture-build.ymlmatches /*.cs, engines/, tests/** and runs another Windows build; several unrelated workflows also match the commonengines/ARIEC61850.lock.jsonor services files, e.g. IO, SCL, installer..github/workflows/ci-p1-shadow.ymlclassifier is advisory and the actual heavy workflows do not consume its classification as a gating contract.- Existing per-PR
cancel-in-progressdoesn't prevent a second wave after first completes. Each PR update can repeat heavy jobs.
Mandatory correction — without sacrificing safety
- No CI-triggering hotfix for this incident while P10.9K is being qualified. Preserve current known-success candidate SHA. No gratuitous new PR/commit/merge just to fix CI.
- Design one cheap fast PR preflight for semantic/lock/parser/source validation and deterministic touched-path classification. The selection policy must be checked by hostile negative fixtures (unknown paths fail-safe).
- Run one canonical Windows build/test + portable package for a frozen SHA when candidate promotion or real-user artifact is needed. Do not launch it on every intermediate commit. Honor repository required checks/rulesets; do not silently disable mandatory branch protection.
- Let dependent IO/SV/COMTRADE/installer workflows consume exact-SHA canonical evidence/artifacts instead of cloning, recompiling and packaging redundantly. Do not allow stale artifact reuse: verify app tree hash, pinned engine/native SHA and test manifest.
- Field Capture and long-running physical/provenance/package checks should be targeted, manual/on-demand at qualification, or cheap evidence validation; not automatically rerun another full build on every control semantic code change. Preserve the strict P10 physical/RCB/GI/no cyclic polling gates.
- Fix per-PR concurrency + event semantics and source-change classification to prevent redundant post-ready/merge PR waves. Audit exact event action before assuming any one action caused wave 2.
- Cost KPI: for unchanged source+engine+native dependency identity, at most one canonical Windows compile/package and no duplicate full Windows build; report workflow counts, runner-minutes and critical-path wall time per candidate. For invalid/ambiguous control: exactly zero MMS Writes remains mandatory.
- Validate proposed CI-routing changes with an offline dry-run of event/path fixtures first. Apply one consolidated workflow PR only after confidence, rather than incrementally triggering the entire workflow fan-out multiple times.
Status and scope
- This is a CI waste defect, not an IEC 61850 functional requirement. P10.9K code CI green is distinct from physical ZIV qualification.
- P10.9K milestone remains Issue P10.9K — generic multi-vendor control semantics, safety and lightweight MMS inventory #540; regression/acceptance authority remains canonical handoff Issue CANONICAL HANDOFF — ARSAS P10.9K code-verified multi-vendor control safety / physical retest pending #525; BCUGE P10.8: isolate BCUGE BRCB BufOvfl and SqNum/EntryID discontinuities with same-slot wire-order evidence #520 remains unresolved.
- Do not change
mainor the verified physical baseline. Any CI correction must preserve exact-source provenance and no unsafe bypass of required acceptance.
Activity
masarray commentedon Oct 12, 2026
P10.9L measurable CI-fanout follow-up (2026-10-12): three successive draft PR #543 consumer heads each triggered 14 pull_request workflow runs despite control-only source/test/lock changes:
c4abd9c16e03acd9b1d529b78a48576086f7e5a0(canonical run #38239310808, failed on initial unqualified engine syntax);5564d01f1a48bc0bdd6a36363713fe6f041717de(canonical #38239824870, passed build but failed 4 regression oracles);7d1467817675a53e619428d5b974fa4dc258c9d1(canonical #38240471645, SUCCESS including 1,654/1,654 ARSAS tests, 19/19 engine tests and portable EXE smoke; artifact #11742369076). Total 42 consumer workflow runs over three waves, including SCL, SV, IO, Installer, GOOSE/FAT-adjacent workflows unrelated to the control patch. Engine PR #169 required separate .NET qualification, SUCCESS at #38239451501. This is quantified evidence, not a workflow fix. Future work must preserve required safety checks but route changed-path control-only work to one canonical Windows test/package and reuse exact-SHA proofs/artifacts; avoid further unqualified broad fanout.masarray commentedon Oct 26, 2026
D18 coordinated CI-cost audit (ARSAS RC
817c3910e8223fa2ac531a4426ada5d2d4bbbe50; no new Actions run): #653 is an unabsorbed, substantive optimization, not a stale duplicate. Its original changes to.github/workflows/build.ymlwould add a fail-closed read-onlypr-scopeclassification for docs-only PRs while always building source/mixed/unknown changes;.github/workflows/smart-discovery-capture-build.ymlwould gate extra Windows capture behindci-field-captureor manual dispatch. None of the keypr-scope,needs.pr-scope.outputs.needs_windows,ci-field-capturecontrols exists in current RC source, and originalscripts/test-ci-workflows.pytest methods are absent. #653 HEAD779bbdef9865004317e8fcf1de282bc0aacdd23fdiverges (behind 5) and should be rebased/surgically reconciled, not merged naively or closed.Separately #677 was retired during D18 because direct EXE Actions artifact upload (upload-artifact@v7, archive:false) and both original regression test blobs exist in RC. That does not reduce duplicate Windows builds by itself, and #660 opt-in public prerelease is a separate security/product approval gate (its workflow is missing in RC).
Next correct P0 CI approach: owner-isolated design review; assert docs-only skip is narrowly allowed and still leaves downstream checks fail-closed; source/engine/security/unknown-file PRs, push and manual always retain exact-SHA Windows proof; protect specialist capture for physical promotion; check required GitHub statuses/branch protection before altering job conditional; validate paths with targeted workflow tests and compare actual runner minutes. Do not launch a fleet of CI runs simply to prove a docs-only audit.
Source/decision checkpoint: https://github.com/masarray/arsas/blob/audit/arsas-reconciliation-d18-20261027/docs/maintenance/ARSAS_RECONCILIATION_D18_2026-10-27.md. This comment records evidence, not a claim #542 is resolved.
masarray commentedon Oct 26, 2026
D19 update on CI COST P0: substantial code change made but not yet merge-qualified. RC-based
ci/p4-failclosed-optimized-gates-d19-20261027, exact6065c6e8e36d67d1f5488153aa99a01183c3cf0d, implements verified-docs-only cheap PR preflight and explicitly labeled/manual Field Capture Windows. Retains canonical Windows for all source/mixed/unknown/API-error/classifier-failure/push/manual paths; old #653 would fail open if the classifier job failed, now fixed. The direct EXE artifact from retired #677 is preserved. Changes: 2 workflows, existing workflow regression test suite, and one audit document; no runtime/engine modifications.Executed 24 offline JS condition simulations. GitHub Actions workflow execution and Python P0 suite have not been run; creating a new PR now could fan out CI, so branch was committed without opening one. Branch protection endpoint returned 403 (no admin scope); repository ruleset list empty, not definitive proof of required-check topology. No real-world CI cost reduction claimed yet; measure representative workload only after owner verifies branch protection and authorizes a scoped qualification. Keep #542 and #653 OPEN; no automatic merge/release/promotion. Full evidence: https://github.com/masarray/arsas/blob/ci/p4-failclosed-optimized-gates-d19-20261027/docs/audits/CI_P4_D19_FAILCLOSED_IMPLEMENTATION_2026-10-27.md.
masarray commentedon Oct 26, 2026
D20 engineering checkpoint: one scoped draft qualification PR now opened, #757, base protected integrated RC
release/arsas-integrated-rc0-20261024, HEADe0c9724b203ac38f2a947d71795ac1173ca5ed5b; not main or stable and not merged. It implements safer CI-P4 on top of original #653, now with a new executable regression test that extracts actual GitHub-script JS and Windows/Field Capture conditions from checked-out workflow files and checks 26 fail-closed test cases using Node VM. It runs within the existing read-onlyCI P0 Workflow Integritysuite, no new workflow/runner introduced.Actual GitHub run evidence for exact candidate: CI P0 Workflow Integrity #40407726690 is SUCCESS, with existing workflow test suite
Ran 12 tests ... OK; Build ARSAS #40407726657 classifier job SUCCESS and real run log saysChanged paths=4, Windows required=true. The canonical Windows job started (not yet certified). Smart Discovery Field Capture #40407726714 skipped both manual-regression and extra Windows build for a draft/unlabeled PR, as intended. CI P1 #40407726721 SUCCESS. Other source-impacting parallel workflows can still run on this PR and remain separate cost topics.Scope and safety: Previous design would fail open on failed classifier; current
build-windowsusesalways(), classifier result and only explicitneeds_windows=falseto skip. No WPF/C# runtime, engine lock, safety, Write/SG7/UI, stable or RC changes. Do not merge #757 until remaining CI is checked, required branch-protection checks manually confirmed (GitHub App read gives HTTP 403), docs-only and label/manual event proof independently evaluated, and measured billed Windows savings. Do not rerun passed CI or push no-op changes to trigger more runs. Legacy #653 remains OPEN as historical reference, not automatically merged or closed.masarray commentedon Oct 26, 2026
D21 qualification — real GitHub Actions evidence, exact SHAs; no reruns or source changes
Parent CI P4 draft #757 remains OPEN DRAFT, exact HEAD
e0c9724b203ac38f2a947d71795ac1173ca5ed5b, base protected RC817c3910e8223fa2ac531a4426ada5d2d4bbbe50, NOT main.Source-impacting real PR wave (one attempt, 8 workflows, all finished):
Changed paths=4, Windows required=true; canonical Windows job successful. Exact SHA producedARSAS-test-evidence,ARSAS-windows-package-input(sealed) and directARSAS-1.6.42-win-x64-portable.exeActions artifacts, plus source/native evidence. These are candidate CI artifacts, not stable release/physical acceptance.Genuine documentation-only canary #758 was created from the candidate HEAD on a separate one-Markdown-file branch, targeted back to that candidate feature branch. Real Build ARSAS #40410571266 SUCCESS,
pr-scopejob SUCCESS (logChanged paths=1, Windows required=false), canonical Windows job SKIPPED, zero artifacts on canary SHA00b0c519107e3d3cd7560428379a22500bb78b9a. Only one workflow triggered, demonstrating real docs-only CI saving without false package proof. Canary CLOSED UNMERGED after writing exact evidence. No commit added to #757 or the protected RC.Qualification now supported: actual source PR Windows admission/build; actual docs-only Windows skip/no sealed artifact; actual Field Capture draft/unlabeled opt-out; P0 26-case policy regression. Still NOT supported: main branch-protection required-check topology (GitHub App GET 403), live nondraft labeled and manual Field Capture dispatch paths, real physical IEC 61850 FAT, or a robust measured cross-PR billable-minute savings baseline.
Disposition: retain #757 as DRAFT and #542 as OPEN, refrain from relabeling it
ci-field-capture(would spend Windows runners) or pushing no-op changes. No release/engine pin/main/safety/Write/SG7/GUI modification.masarray commentedon Oct 26, 2026
D21 workflow fan-out topology — avoid optimizing the wrong runners
The #757 PR triggered eight workflows, but NOT eight Windows builds. GitHub job states and source workflows on exact candidate
e0c9724b203ac38f2a947d71795ac1173ca5ed5bdemonstrate:pull_requestjobs useruns-onconditional selecting ubuntu-latest for PR, consuming the canonical exact-SHA evidence; do not falsely count these as redundant Windows compilations or disable their checks.Changed paths=1, Windows required=false, Windows job SKIPPED, zero artifacts, then was CLOSED UNMERGED.Next CI cost work: inspect whether COMTRADE specialized native proof can safely reuse exact-SHA canonical native CTest and artifact digest on nondomain-impacting workflow changes, without weakening full native validation for COMTRADE/ARDIREC changes. Verify the exact native/engine lock, family, source SHA, proof availability and fail-closed manual controls before modification. Count actual Windows job minutes, not simply workflow events; no billing saving claimed without evidence. #542 remains OPEN.
masarray commentedon Oct 26, 2026
D22 CI-cost audit of the actual #757 source wave found independent COMTRADE CTest/DLL parity with the canonical build: COMTRADE run 40407726689 advisory log reported nine matching native CTest names and the identical native DLL SHA-256
55a4818f45bb43438296c3cbd40e425b23fe21d892bf101f2c82e93a56353986, canonical run40407726657completed SUCCESS. Both compare on GitHub's exact synthetic merge SHA, immutable ARIECf75e323e9f8443aa20e0a3818a1d5d77255b2c0fand ARDIREC7dbc7149db668126a493ce338264363a6ec5ec00. Crucial: canonical comparison returneddeduplicationAuthorized=false,independentWindowsCTestRetained=true,releasePromotionAuthority=false. Do not delete COMTRADE independent Windows CMake/CTest merely because one parity sample matched; its workflow explicitly makes this proof mandatory. D22 instead delivered a unique missing read-only field diagnostic validator in draft #759, routed into existing CI-P0 (20 tests GREEN, no new workflow). A separate new source PR wave is deliberately limited to three workflows, not the eight seen for #757. #542 still open; true native dedup needs multi-run evidence and owner-approved equivalent acceptance policy. No claimed billed-minute savings or weakened native tests.masarray commentedon Oct 27, 2026
D26 CI-P3J provenance hardening — exact PR identity, no deduplication permission
A targeted source fix is staged as draft PR #760, commit
2c95d0c, isolated on protected integrated RC. The advisory COMTRADE native-shadow chooser previously matched canonicalBuild ARSASonly on shared head SHA/branch. Multiple PRs may share those while targeting different bases (e.g. #727 and #729), so the latest other PR run could be mistaken for the correct proof. D26 reads exact PR identity fromGITHUB_EVENT_PATHand filters canonical runpull_requests[].number, still requiring exact source synthetic merge/engine/ArdIrec proof.Five additional hostile offline tests were added (20 total). CI P0 #40463264351 completed SUCCESS (20/20). The independent COMTRADE Windows native job within run #40463264264 also completed SUCCESS; at time of this comment its advisory observer and the canonical Windows #40463264271 were still ongoing. No final parity claim until those complete.
No cost-saving claim from this patch. Independent Windows COMTRADE CMake/CTest stays mandatory;
deduplicationAuthorized=false, release promotion remains false, and no workflows were deleted or weakened. #757 fail-closed routing is not touched. D26 audit: checkpoint.masarray commentedon Oct 27, 2026
D29 exact-scope issue disposition
Targeted CI P4 waste mitigation delivered: #757 merged into integrated RC d459861 (fail-closed docs-only Windows admission and opt-in Field Capture), and #760 exact PR identity on the canonical COMTRADE native shadow. Obsolete duplicate staging/release PR #727/#729 were later closed, limiting duplicated PR-head waves. Scoped CI efficiency work can be marked completed, but this is NOT a measured zero-waste or global runner-minute guarantee. Mandatory independent COMTRADE Windows CMake/CTest remains on; no permission to deduplicate it based on one parity observation. GitHub PR mainline R7/reference guards remain genuine and are not bypassed.
No repeat capture is requested from the operator. This is issue-state reconciliation, not a new hardware/engine release assertion.