Skip to content

[ci] Stop the scheduled builds from rebuilding on every tick, and fix 3.1 - #409

Merged
morningman merged 1 commit into
apache:mainfrom
morningman:ci-fix-scheduled-build-loop
Aug 17, 2026
Merged

[ci] Stop the scheduled builds from rebuilding on every tick, and fix 3.1#409
morningman merged 1 commit into
apache:mainfrom
morningman:ci-fix-scheduled-build-loop

Conversation

@morningman

@morningman morningman commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Problem

The Actions tab is solid red. Over the last 15 hours of scheduled runs:

workflow result
Build (master) 28 / 28 green
Build (4.0) 29 / 29 green
Build (1.2-lts) 0 / 28
Build (2.1) 0 / 28
Build (3.0) 0 / 28
Build (3.1) 0 / 28
Build (2.0) 8 green (build skipped) / 20 red

That is a handful of real failures amplified by one state machine bug.

The amplifier

The pipeline keeps its entire state in the body of the automation* release.
prerelease reads Doris Version: out of it to decide whether anything needs
building — but the failure job overwrote the note with only Status: *FAILURE*
plus a checksum block, dropping that line:

# success job -- preserves the version
content="$(gh release view "${tag_name}" | sed -n '/Update Time:/,/Doris Version:/p')"
echo -ne "${content}\nStatus: *SUCCESS*\n..." >release_note.md

# failure job -- overwrites the whole note, version is gone
echo -ne "Status: *FAILURE*\n..." >release_note.md

Thirty minutes later prerelease read an empty version, logged The first release was detected and rebuilt everything, which failed again and wiped the
version again. Straight from a run this morning:

Last Version:
Current Version: a3052de2e5cbc397d67b9cd9d5cd9701aaa6426c

branch-1.2-lts shows how bad this got: no commits in years, still running a
full macOS build every 30 minutes.

manual-build.yml had the same failure job and writes to the same
automation release as the scheduled master build, so one red manual build
could drop master into the loop too.

The real failures underneath

  • 2.1 / 3.0 / 3.1LZO2_DOWNLOAD in those branches still points at
    https://fossies.org/linux/misc/lzo-2.10.tar.gz, which now answers 410
    Gone
    . master and 4.0 already moved to oberhumer.com. 3.1 additionally
    shares 4.0's dead libuuid mirror, nchc.dl.sourceforge.net.
  • 1.2-lts — curl 7.79.0 no longer builds against the libngtcp2 1.24.0 on
    the macOS runner image: configure finds ngtcp2, does not find
    libngtcp2_crypto_openssl (renamed upstream), enables it anyway, and
    vquic/ngtcp2.c dies on the missing header.
  • 2.0 — the Linux job ran out of disk: fatal error: error writing to /tmp/ccAeBk6L.s: No space left on device while building aws-sdk-cpp.

Changes

State machine

  • failure records the version it tried to build, exactly like success does,
    so the state can no longer be lost.
  • That removes the accidental retry the loop was providing, so prerelease
    gains a bounded one: a run whose predecessor ended in FAILURE retries up to
    3 times, then stops. A note with no parseable counter counts as out of
    attempts, so a malformed note can never re-open the loop.
  • failure is gated on needs.prerelease.result == 'success' so it never
    writes an empty version.

Scheduling

  • 1.2-lts, 2.0, 2.1 and 3.0 become workflow_dispatch only — no more polling.
    Trigger them from the Actions tab when a build is needed. Their build and
    success jobs now honour force_build so a manual run actually builds.
  • concurrency group per workflow. A full build takes hours while the schedule
    fires every 30 minutes, so runs were overlapping and racing on the same tag.

3.1, so it can go green

  • lzo and libuuid are pre-seeded in prerelease and repaired in build, the
    same way build-4.0.yml already does it. No-ops once the upstream fix lands,
    since the checksums are identical.
  • fail-fast: false. A single red platform used to cancel the other two, so
    nobody could see whether Linux or Intel macOS were healthy.
  • Drops easimon/maximize-build-space. It reserves only root-reserve-mb on
    / and gcc writes its temporaries to /tmp on / — that is how 2.0 died.
    master and 4.0 already build fine on the stock runner disk.

General hardening

  • prerelease moves off macos-14 for master and 3.1. Without GNU md5sum,
    download-thirdparty.sh prints Warn: md5sum is not installed, skips
    verification entirely and packs the 0-byte files failed downloads leave behind
    into the source tarball — so the build jobs fail on a checksum prerelease
    never looked at.
  • gh release delete-asset ... --yes before upload. --yes matters: without it
    gh cannot confirm the prompt on a runner, so the existing delete-asset in
    build-4.0.yml silently did nothing and left the plain upload to fail.
  • actions/checkout v4 to v5, clearing the Node 20 deprecation warning on every
    run.

Upstream

Companion PRs fix the mirrors at the source in apache/doris:

  • branch-3.1: lzo + libuuid
  • branch-4.0: libuuid

Testing

  • Every workflow parses as YAML and every run: block passes bash -n.
  • The rewritten check_diff state machine was extracted from build-3.1.yml
    and run against stubbed gh/git over 11 cases: unchanged HEAD, new HEAD
    with and without thirdparty/ changes, the three retry steps, exhaustion at
    attempt 3, an unparseable counter, and a replay of the old broken note showing
    the loop now terminates.

Relation to #405

Rebased on top of #405, which landed while this was being written and fixed the
same failure job in build.yml by reading Doris Version back out of the note
and failing the job when it is missing.

That protects an intact note, but it cannot repair one that has already lost the
line — and a note in that state is precisely what keeps the rebuild loop
running: content comes back empty, the job exits 1, the note stays broken, and
the next tick rebuilds again. Four of the five automation* releases are in that
state right now.

So the resolution takes the version from the prerelease job's output instead of
re-parsing the note. It is always available, needs no guard, and recovers a note
that is already broken. Everything else from #405 is untouched, and the same fix
is now applied to the six workflows #405 did not cover — including
manual-build.yml, which writes to master's automation release and could
therefore drop master into the loop on its own.

🤖 Generated with Claude Code

… 3.1

The Actions tab has been solid red: 1.2-lts, 2.1, 3.0 and 3.1 failed 28 out of
28 scheduled runs over the last 15 hours, and 2.0 failed 20 out of 28. That is
one failure pattern amplified by one state machine bug.

The bug: the whole pipeline keeps its state in the body of the `automation*`
release. The prerelease job reads `Doris Version:` out of it to decide whether
anything needs building, but the failure job overwrote the note with just
`Status: *FAILURE*` and a checksum block, dropping that line. Thirty minutes
later prerelease read an empty version, concluded "The first release was
detected" and rebuilt everything from scratch, which failed again and wiped the
version again. branch-1.2-lts shows how bad this got: that branch has had no
commits in years and was still running a full macOS build every 30 minutes.

The failure job now records the version it tried to build, exactly like the
success job does, so the state can no longer be lost. Because that also removes
the accidental retry the loop was providing, the prerelease job gains a bounded
one: a run whose predecessor ended in FAILURE retries up to 3 times and then
stops. A note without a parseable counter is treated as out of attempts, so a
malformed note can never re-open the loop.

manual-build.yml had the same failure job and writes to the same `automation`
release as the scheduled master build, so one manual build that went red used to
put master into that loop as well.

What was actually failing underneath:

  - 2.1/3.0/3.1: LZO2_DOWNLOAD in those branches still points at
    fossies.org/linux/misc/lzo-2.10.tar.gz, which now answers 410 Gone. master
    and 4.0 already moved to oberhumer.com. 3.1 also shares 4.0's dead libuuid
    mirror, nchc.dl.sourceforge.net. Both are pre-seeded in build-3.1.yml the
    same way 4.0 already does it, and fixed at the source in apache/doris.
  - 1.2-lts: curl 7.79.0 no longer builds against the libngtcp2 that ships on
    the macOS runner image (ngtcp2_crypto_openssl.h was renamed upstream).
  - 2.0: the Linux job ran out of disk in /tmp while building aws-sdk-cpp.

1.2-lts, 2.0, 2.1 and 3.0 are switched to workflow_dispatch only. They are no
longer polled; trigger them from the Actions tab when a build is needed.

Other hardening:

  - fail-fast: false on 3.1. A single red platform used to cancel the other two,
    so nobody could see whether Linux or Intel macOS were healthy.
  - concurrency group per workflow. A full build takes hours while the schedule
    fires every 30 minutes, so runs were overlapping and racing on the same
    release tag.
  - 3.1 drops easimon/maximize-build-space. It reserves only root-reserve-mb on
    /, and gcc writes its temporaries to /tmp on /, which is how 2.0 died. master
    and 4.0 already build on the stock disk.
  - prerelease moved off macos-14 for master and 3.1. Without GNU md5sum,
    download-thirdparty.sh skips verification entirely and packs the 0-byte
    files that failed downloads leave behind into the source tarball, so the
    build jobs fail on a checksum prerelease never looked at.
  - gh release delete-asset before upload, with --yes so it actually runs
    non-interactively.
  - actions/checkout v4 -> v5, clearing the Node 20 deprecation warning.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@morningman
morningman force-pushed the ci-fix-scheduled-build-loop branch from cbc8079 to b506e41 Compare August 17, 2026 03:23
@morningman
morningman merged commit 8078e1e into apache:main Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants