[ci] Stop the scheduled builds from rebuilding on every tick, and fix 3.1 - #409
Merged
Merged
Conversation
… 3.1
The Actions tab has been solid red: 1.2-lts, 2.1, 3.0 and 3.1 failed 28 out of
28 scheduled runs over the last 15 hours, and 2.0 failed 20 out of 28. That is
one failure pattern amplified by one state machine bug.
The bug: the whole pipeline keeps its state in the body of the `automation*`
release. The prerelease job reads `Doris Version:` out of it to decide whether
anything needs building, but the failure job overwrote the note with just
`Status: *FAILURE*` and a checksum block, dropping that line. Thirty minutes
later prerelease read an empty version, concluded "The first release was
detected" and rebuilt everything from scratch, which failed again and wiped the
version again. branch-1.2-lts shows how bad this got: that branch has had no
commits in years and was still running a full macOS build every 30 minutes.
The failure job now records the version it tried to build, exactly like the
success job does, so the state can no longer be lost. Because that also removes
the accidental retry the loop was providing, the prerelease job gains a bounded
one: a run whose predecessor ended in FAILURE retries up to 3 times and then
stops. A note without a parseable counter is treated as out of attempts, so a
malformed note can never re-open the loop.
manual-build.yml had the same failure job and writes to the same `automation`
release as the scheduled master build, so one manual build that went red used to
put master into that loop as well.
What was actually failing underneath:
- 2.1/3.0/3.1: LZO2_DOWNLOAD in those branches still points at
fossies.org/linux/misc/lzo-2.10.tar.gz, which now answers 410 Gone. master
and 4.0 already moved to oberhumer.com. 3.1 also shares 4.0's dead libuuid
mirror, nchc.dl.sourceforge.net. Both are pre-seeded in build-3.1.yml the
same way 4.0 already does it, and fixed at the source in apache/doris.
- 1.2-lts: curl 7.79.0 no longer builds against the libngtcp2 that ships on
the macOS runner image (ngtcp2_crypto_openssl.h was renamed upstream).
- 2.0: the Linux job ran out of disk in /tmp while building aws-sdk-cpp.
1.2-lts, 2.0, 2.1 and 3.0 are switched to workflow_dispatch only. They are no
longer polled; trigger them from the Actions tab when a build is needed.
Other hardening:
- fail-fast: false on 3.1. A single red platform used to cancel the other two,
so nobody could see whether Linux or Intel macOS were healthy.
- concurrency group per workflow. A full build takes hours while the schedule
fires every 30 minutes, so runs were overlapping and racing on the same
release tag.
- 3.1 drops easimon/maximize-build-space. It reserves only root-reserve-mb on
/, and gcc writes its temporaries to /tmp on /, which is how 2.0 died. master
and 4.0 already build on the stock disk.
- prerelease moved off macos-14 for master and 3.1. Without GNU md5sum,
download-thirdparty.sh skips verification entirely and packs the 0-byte
files that failed downloads leave behind into the source tarball, so the
build jobs fail on a checksum prerelease never looked at.
- gh release delete-asset before upload, with --yes so it actually runs
non-interactively.
- actions/checkout v4 -> v5, clearing the Node 20 deprecation warning.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
morningman
force-pushed
the
ci-fix-scheduled-build-loop
branch
from
August 17, 2026 03:23
cbc8079 to
b506e41
Compare
hello-stephen
approved these changes
Aug 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Actions tab is solid red. Over the last 15 hours of scheduled runs:
That is a handful of real failures amplified by one state machine bug.
The amplifier
The pipeline keeps its entire state in the body of the
automation*release.prereleasereadsDoris Version:out of it to decide whether anything needsbuilding — but the
failurejob overwrote the note with onlyStatus: *FAILURE*plus a checksum block, dropping that line:
Thirty minutes later
prereleaseread an empty version, loggedThe first release was detectedand rebuilt everything, which failed again and wiped theversion again. Straight from a run this morning:
branch-1.2-ltsshows how bad this got: no commits in years, still running afull macOS build every 30 minutes.
manual-build.ymlhad the samefailurejob and writes to the sameautomationrelease as the scheduled master build, so one red manual buildcould drop master into the loop too.
The real failures underneath
LZO2_DOWNLOADin those branches still points athttps://fossies.org/linux/misc/lzo-2.10.tar.gz, which now answers 410Gone. master and 4.0 already moved to
oberhumer.com. 3.1 additionallyshares 4.0's dead libuuid mirror,
nchc.dl.sourceforge.net.libngtcp2 1.24.0onthe macOS runner image:
configurefinds ngtcp2, does not findlibngtcp2_crypto_openssl(renamed upstream), enables it anyway, andvquic/ngtcp2.cdies on the missing header.fatal error: error writing to /tmp/ccAeBk6L.s: No space left on devicewhile building aws-sdk-cpp.Changes
State machine
failurerecords the version it tried to build, exactly likesuccessdoes,so the state can no longer be lost.
prereleasegains a bounded one: a run whose predecessor ended in
FAILUREretries up to3 times, then stops. A note with no parseable counter counts as out of
attempts, so a malformed note can never re-open the loop.
failureis gated onneeds.prerelease.result == 'success'so it neverwrites an empty version.
Scheduling
workflow_dispatchonly — no more polling.Trigger them from the Actions tab when a build is needed. Their
buildandsuccessjobs now honourforce_buildso a manual run actually builds.concurrencygroup per workflow. A full build takes hours while the schedulefires every 30 minutes, so runs were overlapping and racing on the same tag.
3.1, so it can go green
prereleaseand repaired inbuild, thesame way
build-4.0.ymlalready does it. No-ops once the upstream fix lands,since the checksums are identical.
fail-fast: false. A single red platform used to cancel the other two, sonobody could see whether Linux or Intel macOS were healthy.
easimon/maximize-build-space. It reserves onlyroot-reserve-mbon/and gcc writes its temporaries to/tmpon/— that is how 2.0 died.master and 4.0 already build fine on the stock runner disk.
General hardening
prereleasemoves offmacos-14for master and 3.1. Without GNUmd5sum,download-thirdparty.shprintsWarn: md5sum is not installed, skipsverification entirely and packs the 0-byte files failed downloads leave behind
into the source tarball — so the build jobs fail on a checksum
prereleasenever looked at.
gh release delete-asset ... --yesbefore upload.--yesmatters: without itgh cannot confirm the prompt on a runner, so the existing
delete-assetinbuild-4.0.ymlsilently did nothing and left the plain upload to fail.actions/checkoutv4 to v5, clearing the Node 20 deprecation warning on everyrun.
Upstream
Companion PRs fix the mirrors at the source in
apache/doris:branch-3.1: lzo + libuuidbranch-4.0: libuuidTesting
run:block passesbash -n.check_diffstate machine was extracted frombuild-3.1.ymland run against stubbed
gh/gitover 11 cases: unchanged HEAD, new HEADwith and without
thirdparty/changes, the three retry steps, exhaustion atattempt 3, an unparseable counter, and a replay of the old broken note showing
the loop now terminates.
Relation to #405
Rebased on top of #405, which landed while this was being written and fixed the
same
failurejob inbuild.ymlby readingDoris Versionback out of the noteand failing the job when it is missing.
That protects an intact note, but it cannot repair one that has already lost the
line — and a note in that state is precisely what keeps the rebuild loop
running:
contentcomes back empty, the job exits 1, the note stays broken, andthe next tick rebuilds again. Four of the five
automation*releases are in thatstate right now.
So the resolution takes the version from the prerelease job's output instead of
re-parsing the note. It is always available, needs no guard, and recovers a note
that is already broken. Everything else from #405 is untouched, and the same fix
is now applied to the six workflows #405 did not cover — including
manual-build.yml, which writes to master'sautomationrelease and couldtherefore drop master into the loop on its own.
🤖 Generated with Claude Code