Skip to content

[ci] Unblock Docker on the Linux build, cache ccache, and stop macOS from stalling - #410

Merged
morningman merged 1 commit into
apache:mainfrom
morningman:ci-decouple-docker-and-cache
Aug 17, 2026
Merged

[ci] Unblock Docker on the Linux build, cache ccache, and stop macOS from stalling#410
morningman merged 1 commit into
apache:mainfrom
morningman:ci-decouple-docker-and-cache

Conversation

@morningman

Copy link
Copy Markdown
Contributor

What this changes

Run 31988123966
took 5h01m end to end. Per-job:

job wall clock
Build (Linux-arm64) 1h27m54s
Build (macOS-arm64) 1h51m06s
Build (Linux) 2h45m49s
Build (macOS-x86_64) 4h22m41s
Update Docker Image 25m24s, started 06:58

Four things fall out of that, and this PR addresses each.

1. The Docker image waited 1h39m for nothing

update-docker only consumes doris-thirdparty-prebuilt-linux-x86_64.tar.xz,
which the Linux job published at 05:19. But as needs: build it depended on
the whole matrix, so it did not start until 06:58 - held up by macOS x86_64,
which finished at 06:56.

A matrix job cannot be depended on per-leg, so the build job becomes a reusable
workflow (.github/workflows/build-target.yml) called once per platform.
update-docker now depends on build-linux-x86_64 alone. The provenance check it
already runs against the archive is unchanged.

Splitting this way is also what makes items 3 and 4 below expressible per
platform. The package lists and the ldb-toolchain choice, which the matrix entries
used to carry, are derived from RUNNER_OS / RUNNER_ARCH inside the reusable
workflow - the two Linux entries and the two macOS entries were identical except
for the toolchain script.

2. Every macOS build sat idle for exactly 10 minutes

thrift's configure finds the dotnet the runner images ship, so make runs
dotnet build -c Release for lib/netstd. The build itself finishes in seconds
(Build succeeded, 12s on macos-14, 46s on macos-15-intel), and then the log goes
completely silent before Making install in compiler/cpp:

runner gap
macos-14 02:56:58.63 -> 03:06:58.66 = 600.03s
macos-15-intel 04:00:38.09 -> 04:10:36.88 = 598.79s
ubuntu-22.04 0.01s

An exact 10 minutes on both Macs is not compute, it is the compiler server that
dotnet build leaves behind holding the stdout it inherited for its full
keep-alive. UseSharedCompilation=false and MSBUILDDISABLENODEREUSE=1 stop it
from being started at all. 10 minutes back on each macOS job.

3. macOS x86_64 is 87% of the pipeline's wall clock

macos-15-intel is roughly 2x slower than the M1 runner at compiling
(lance-c 41m43s vs 22m19s, grpc 2.1x, arrow 2.3x) and about 8x slower at
spawning processes - which is what a tree of 70+ autotools packages does all day.
Identical configure scripts, same probe count:

package macos-14 ubuntu-22.04-arm macos-15-intel
curl configure (576 probes) 21s 19s 178s
openssl (~6500 log lines) 43s 34s 373s

It already runs with continue-on-error, so it is now built only once its
published archive is older than vars.MACOS_X86_MAX_AGE_DAYS (default 7)
rather than on every thirdparty change. The prerelease job reads the asset's
updatedAt off the release, so this self-regulates with no extra cron.
workflow_dispatch with force_build still builds it unconditionally, and the
threshold is a repository variable so it can be retuned without a code change.

4. Nothing was cached between runs

ccache is installed on every runner, but nothing routed compilation through it
and nothing persisted it, so a thirdparty change touching one package still
rebuilt all ~80 from scratch. This sets ENABLE_THIRDPARTY_CCACHE=ON and carries
$CCACHE_DIR across runs with actions/cache (CCACHE_MAXSIZE=2G compressed,
four platforms inside the repository's 10 GB Actions cache budget).

The cache is saved even when the build fails, which is the point: the
prerelease job retries a failed build up to max_attempts=3, and today a flaky
download mirror can cost three consecutive four-hour builds.

This half only does something once apache/doris honours
ENABLE_THIRDPARTY_CCACHE (apache/doris#66839). Until then it is an unused
environment variable and the cache stays empty - the workflow is safe to merge in
either order.

Also

success and failure now list every build job. success had to move to
!cancelled() && !failure(): build-macos-x86_64 is skipped on most runs, and a
skipped dependency would otherwise skip success too, leaving the release note
stuck at BUILDING - which is precisely the state that stops all future scheduled
runs from doing anything.

Testing

actionlint is clean on both files. The behaviour itself can only be exercised by
running the pipeline; the reusable workflow is a faithful move of the existing
steps, with matrix.config.name / matrix.config.os replaced by inputs.target /
RUNNER_OS / RUNNER_ARCH, plus the ccache and dotnet environment above.

Worth a workflow_dispatch run with force_build=true before relying on it, so
that all four platforms and the Docker job are exercised in one go.

…from stalling

Run 31988123966 took 5h01m end to end. Where it went, and what this changes.

**The Docker image waited 1h39m for nothing.** `update-docker` only consumes
`doris-thirdparty-prebuilt-linux-x86_64.tar.xz`, which was published at 05:19,
but as `needs: build` it depended on the whole matrix and did not start until
06:58 - held up by macOS x86_64, which finished at 06:56. The build job is now a
reusable workflow called once per platform, so `update-docker` can depend on
`build-linux-x86_64` alone. Splitting it this way is also what makes the two
changes below expressible per platform.

**Every macOS build sat idle for exactly 10 minutes.** thrift's configure finds
the dotnet the runner images ship, so `make` runs `dotnet build -c Release` for
`lib/netstd`. The build itself finishes in seconds, and then the compiler server
it leaves behind holds the stdout it inherited for its full keep-alive while
nothing happens - 600.0s on macos-14, 598.8s on macos-15-intel, 0.01s on Linux.
Set `UseSharedCompilation=false` and `MSBUILDDISABLENODEREUSE=1` so no server is
left running.

**macOS x86_64 costs 4h22m41s of a 5h01m run** - against 1h27m54s for Linux
arm64 - because that runner is roughly 2x slower than the M1 one at compiling and
about 8x slower at spawning processes, which is what a tree of 70+ autotools
packages does all day. curl's 576 configure probes take 21s on macos-14 and 178s
there. It already runs with `continue-on-error`, so build it only once its
published archive is older than `vars.MACOS_X86_MAX_AGE_DAYS` (default 7) rather
than on every thirdparty change. `workflow_dispatch` with `force_build` still
builds it unconditionally.

**Nothing was cached between runs.** ccache is installed on the runners but
nothing routed through it and nothing persisted it, so a thirdparty change that
touches one package still rebuilt all ~80. Set `ENABLE_THIRDPARTY_CCACHE=ON` and
carry `$CCACHE_DIR` across runs with `actions/cache`. The cache is saved even
when the build fails, which is what makes the `max_attempts=3` retry in the
prerelease job affordable - today a flaky download mirror can cost three full
builds. This needs apache/doris to honour `ENABLE_THIRDPARTY_CCACHE`; until that
lands it is an unused variable and the cache stays empty.

`success` and `failure` now list every build job. `success` had to move to
`!cancelled() && !failure()`, because a skipped `build-macos-x86_64` would
otherwise skip it too and leave the release note stuck at BUILDING - which is the
state that stops all future scheduled runs.
@morningman
morningman merged commit 7adcd59 into apache:main Aug 17, 2026
morningman added a commit to morningman/doris that referenced this pull request Aug 17, 2026
…nd cache ccache

Two of the four fixes in apache/doris-thirdparty#410 apply to this workflow too.

thrift's configure finds the dotnet the runner images ship, so `make` runs
`dotnet build -c Release` for lib/netstd. The build itself finishes in seconds and
then the compiler server it leaves behind holds the stdout it inherited for its
full keep-alive while nothing happens. Between "Build succeeded" and "Making
install in compiler/cpp" in apache/doris-thirdparty run 31988123966: 600.03s on
macos-14, 598.79s on macos-15-intel, 0.01s on Linux. Both macOS jobs here pay it.
`UseSharedCompilation=false` and `MSBUILDDISABLENODEREUSE=1` stop the server from
being started.

And nothing was cached between runs: ccache is installed on every runner but
nothing routed compilation through it, so a change to one package recompiled all
~80, at `-j 2` on two of the three jobs. Turn on ENABLE_THIRDPARTY_CCACHE and
carry the cache with the ccache-action this repository already vendors.

max-size is 1G per job so that all three together stay inside what is left of the
repository's 10 GB Actions cache budget next to the 5G BE-UT-macOS cache. Note
that a cache saved by a pull request run is only visible to that same pull
request, so this pays off across pushes to one branch; sharing it across pull
requests would need this workflow to run on master.

The other two fixes in that PR do not apply here: there is no Docker job to
decouple, and no macOS x86_64 job to build less often.
morningman added a commit to morningman/doris that referenced this pull request Aug 18, 2026
…nd cache ccache

Two of the four fixes in apache/doris-thirdparty#410 apply to this workflow too.

thrift's configure finds the dotnet the runner images ship, so `make` runs
`dotnet build -c Release` for lib/netstd. The build itself finishes in seconds and
then the compiler server it leaves behind holds the stdout it inherited for its
full keep-alive while nothing happens. Between "Build succeeded" and "Making
install in compiler/cpp" in apache/doris-thirdparty run 31988123966: 600.03s on
macos-14, 598.79s on macos-15-intel, 0.01s on Linux. Both macOS jobs here pay it.
`UseSharedCompilation=false` and `MSBUILDDISABLENODEREUSE=1` stop the server from
being started.

And nothing was cached between runs: ccache is installed on every runner but
nothing routed compilation through it, so a change to one package recompiled all
~80, at `-j 2` on two of the three jobs. Turn on ENABLE_THIRDPARTY_CCACHE and
carry the cache with the ccache-action this repository already vendors.

max-size is 1G per job so that all three together stay inside what is left of the
repository's 10 GB Actions cache budget next to the 5G BE-UT-macOS cache. Note
that a cache saved by a pull request run is only visible to that same pull
request, so this pays off across pushes to one branch; sharing it across pull
requests would need this workflow to run on master.

The other two fixes in that PR do not apply here: there is no Docker job to
decouple, and no macOS x86_64 job to build less often.
morningman added a commit to apache/doris that referenced this pull request Aug 19, 2026
… and slim it down, and add ccache (#66842)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: apache/doris-thirdparty#410

Problem Summary:

Third-party build changes, all measured against
[apache/doris-thirdparty run
31988123966](https://github.com/apache/doris-thirdparty/actions/runs/31988123966).
Supersedes #66837 and #66839, which are now closed.

**First half of a two-PR split.** This PR only changes what the
third-party build
produces (`thirdparty/**` and `.github/workflows/build-thirdparty.yml`).
The consumer
side - `cloud/`, `be/src/io/hdfs_builder.cpp`, `build.sh`,
`run-be-ut.sh`,
`run-cloud-ut.sh`, the thirdparty lifecycle test and the `env.sh` azure
default - is
in #66908, which has to land **after** this one and after the prebuilt
archives have
been rebuilt from it. See "Merge order" at the bottom.

---

## 1. Only build hadoop-libs 3.4.2.4

The tree carried two libhdfs builds: `build_hadoop_libs` built the
**3.3.6.6** fork
into `installed/{include,lib}/hadoop_hdfs/`, `build_hadoop_libs_3_4`
built the
**3.4.2.4** fork into `.../hadoop_hdfs_3_4/`, and on Linux both ran -
11m38s of the
x86_64 build and 10m00s of the arm64 one, for a fork only the cloud
module still
consumed.

The two prefixes were also mixed up on the consumer side:
`be/src/io/fs/hdfs.h` and
`be/CMakeLists.txt` were on 3.4.2.4, while `be/src/io/hdfs_builder.cpp`
and all of
`cloud/` were on 3.3.6.6. Both headers share the `LIBHDFS_HDFS_H`
include guard, so
`hdfs_builder.cpp` compiled against the 3.3.6.6 header while BE linked
the 3.4.2.4
archive. Straightening that out is #66908.

3.4.2.4 stays where it is, under `hadoop_hdfs_3_4/`, so a build-env
image from this
change can still compile older Doris branches. The 3.3.6.6 build and its
`hadoop_hdfs/` prefix are gone, and `hadoop_libs_3_4` now runs on Linux
and macOS
alike.

## 2. Build azure everywhere, and stop building what Doris does not link

**It was only ever built on x86_64 Linux.** `env.sh` forced
`DISABLE_BUILD_AZURE=ON`
on aarch64 and macOS, and `build_azure` skipped `Darwin` a second time,
so BE and the
cloud meta-service lost `+AZURE_BLOB` and `+AZURE_STORAGE_VAULT` there.
Nothing in the
recipe was genuinely x86_64-Linux-only, just two details: `-ldl` fails
on Apple (there
is no libdl; those symbols are in libSystem), and vcpkg ships no
prebuilt cmake/ninja/
curl for aarch64 Linux so it needs `VCPKG_FORCE_SYSTEM_BINARIES=1`. The
triplet is now
derived from `uname` rather than left to vcpkg's host detection, and
`build_azure` no
longer skips Darwin.

`DISABLE_BUILD_AZURE` was answering two questions at once: whether the
third-party
tree *carries* azure, and whether BE and the cloud meta-service *link*
it. The archive
has to grow the libraries before anything can link them, so the two are
split here -
the third-party build reads `DISABLE_THIRDPARTY_BUILD_AZURE` and leaves
`env.sh`'s
switch to the consumers, whose default still skips azure on aarch64 and
macOS until
#66908.

**It was the most expensive package in the tree** - 26m06s of the 2h43m
x86_64 Linux
build - almost none of it for Doris:

| phase | time |
|---|---|
| `vcpkg install` | 22m11s |
|   - protobuf 5.29.3 | 12m29s |
|   - opentelemetry-cpp | 4m35s |
|   - abseil | 1m58s |
|   - openssl 1.1.1n | 1m38s |
|   - curl / libxml2 / zlib / uAMQP / utf8-range | ~1m30s |
| azure SDK configure + build | ~3m55s |

protobuf, abseil and utf8-range were pulled in **only** because
`vcpkg.json` declared
`opentelemetry-cpp` unconditionally - roughly 19 of the 26 minutes.
Doris does not use
azure's OpenTelemetry tracing. On top of that vcpkg builds every port
twice
(`Building x64-linux-dbg` then `-rel`) and Doris links only the release
halves, and the
SDK builds appconfiguration, attestation, eventhubs, keyvault, tables,
template, uAMQP,
storage-files-datalake, storage-files-shares and storage-queues, while
`be/cmake/thirdparty.cmake` links exactly four targets: `azure-core`,
`azure-identity`,
`azure-storage-blobs`, `azure-storage-common`.

So the patch drops `opentelemetry-cpp` and the uAMQP C libraries from
`vcpkg.json`,
flips `DISABLE_AMQP` / `DISABLE_AZURE_CORE_OPENTELEMETRY` to `ON` and
trims the
sub-projects to the four Doris links; `build_azure` writes an overlay
triplet that sets
`VCPKG_BUILD_TYPE release`. What is left for vcpkg to build is curl,
libxml2, openssl
1.1.1n and zlib, release only.

## 3. ccache for the third-party build

`ENABLE_THIRDPARTY_CCACHE=ON` (off by default) exports
`CMAKE_C_COMPILER_LAUNCHER` /
`CMAKE_CXX_COMPILER_LAUNCHER`, which CMake initialises from the
environment, so no
cmake invocation in the script changes. It deliberately does not prefix
`CC`/`CXX`:
CMake splits `"ccache clang"` into the compiler plus a
`CMAKE_<LANG>_FLAGS` entry, and
that leaks into whatever a package exports. Autotools packages are left
alone.

`.github/workflows/build-thirdparty.yml` turns it on and carries the
cache with the
`ccache-action` this repository already vendors. Today a change to one
package
recompiles all ~80, at `-j 2` on two of the three jobs.

`max-size` is **1G** per job so the three together stay inside what is
left of the
repository's 10 GB Actions cache budget next to the 5G `BE-UT-macOS`
cache. Note that a
cache saved by a pull request run is only visible to that same pull
request, so this
pays off across pushes to one branch; sharing it across pull requests
would need this
workflow to run on master, the way `be-ut-mac.yml` uses a schedule for
exactly that
reason. Happy to drop this hunk if the cache budget is too tight.

## 4. Retry a failed vcpkg download instead of losing the azure build

All three jobs of run 32032837067 built 73 of the 74 packages and then
failed on the
74th, azure, in the same place: vcpkg downloading
`github.com/madler/zlib` - HTTP 429
on Linux and macOS, 500 on macOS-arm64, while the Actions cache service
was also
answering "Our services aren't available right now". azure is the only
package that
fetches its own sources (everything else goes through
`download-thirdparty.sh`, which
has a mirror and a fallback), its own retry is three attempts inside one
second, and it
is the last package built - so each job threw away a finished tree over
one file. And
the exposure just tripled now that azure builds on all three platforms.

So the configure is retried with a backoff long enough for a rate-limit
window to pass
(2, 4, 6, 8 minutes) and **only** when the failure was a download - a
port that will not
compile still stops on the first attempt. Ports that did build come back
from vcpkg's
binary cache, and `VCPKG_DOWNLOADS` now points outside the directory
`build_azure` wipes
on entry. `thirdparty/test/azure-vcpkg-retry-test.sh` drives
`build_azure` against a
stubbed cmake and covers both halves, in seconds, in the script-test
job.

## Also: the macOS jobs stalled for 10 minutes each

Carried over from apache/doris-thirdparty#410. thrift's configure finds
the dotnet the
runner images ship, so `make` runs `dotnet build -c Release` for
`lib/netstd`. The build
finishes in seconds and then the compiler server it leaves behind holds
the stdout it
inherited for its full keep-alive while nothing happens:

| runner | "Build succeeded" -> "Making install in compiler/cpp" |
|---|---|
| macos-14 | **600.03s** |
| macos-15-intel | **598.79s** |
| ubuntu-22.04 | 0.01s |

`UseSharedCompilation=false` and `MSBUILDDISABLENODEREUSE=1` stop the
server from being
started. Both macOS jobs in this workflow pay this today.

The other two fixes in apache/doris-thirdparty#410 do not apply here:
there is no Docker
job to decouple, and no macOS x86_64 job to build less often.

## Merge order

1. **This PR.** Its own CI is unaffected by either half of the split:
`env.sh` is
untouched, so BE still links no azure on macOS, and `build.sh` still
agrees with the
   lifecycle test on the `hadoop_hdfs/` sentinel.
2. The apache/doris-thirdparty automation rebuilds the prebuilt
archives. It triggers on
`^thirdparty/` and takes ~3h; only then do the darwin-arm64 and
linux-aarch64
archives carry azure. The aarch64 build-env image has to be rebuilt too.
3. **#66908.**

Between 1 and 3, a Linux tree that picks up a freshly built third-party
archive will not
find `installed/lib/hadoop_hdfs/native/libhdfs.a`, which is still what
`build.sh`'s
`LAST_THIRDPARTY_LIB` looks for, and will start a full third-party
rebuild. Nothing in
CI consumes such an archive for a BE or cloud build - `be-ut-mac.yml` is
on Darwin,
where the sentinel is `libbrotlienc.a`, and `build-extension.yml` only
takes `thrift`
out of the Linux archive - but that is the window #66908 closes.

### Release note

None

### Check List (For Author)

- Test
    - [x] Manual test (add detailed scripts or steps below)

This PR touches `thirdparty/**`, so its own `Build Third Party
Libraries` run
builds the whole tree on ubuntu-22.04, macos-15 and macos-14 - which is
exactly
where the hadoop change, the azure slimming and, for the first time,
azure on
macOS arm64 get exercised. **aarch64 Linux is the one path this workflow
does not
      cover**; that is only built in apache/doris-thirdparty.

      The azure patch was verified to apply cleanly to a pristine
`azure-sdk-for-cpp-azure-core_1.16.0` tarball, and the resulting
`vcpkg.json`
parses and declares only `curl`, `libxml2`, `openssl`, `wil`.
`actionlint` is
clean on the workflow, and the four `thirdparty/test/*.sh` scripts the
script-test
      job runs pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants