[ci] Unblock Docker on the Linux build, cache ccache, and stop macOS from stalling - #410
Merged
Merged
Conversation
…from stalling Run 31988123966 took 5h01m end to end. Where it went, and what this changes. **The Docker image waited 1h39m for nothing.** `update-docker` only consumes `doris-thirdparty-prebuilt-linux-x86_64.tar.xz`, which was published at 05:19, but as `needs: build` it depended on the whole matrix and did not start until 06:58 - held up by macOS x86_64, which finished at 06:56. The build job is now a reusable workflow called once per platform, so `update-docker` can depend on `build-linux-x86_64` alone. Splitting it this way is also what makes the two changes below expressible per platform. **Every macOS build sat idle for exactly 10 minutes.** thrift's configure finds the dotnet the runner images ship, so `make` runs `dotnet build -c Release` for `lib/netstd`. The build itself finishes in seconds, and then the compiler server it leaves behind holds the stdout it inherited for its full keep-alive while nothing happens - 600.0s on macos-14, 598.8s on macos-15-intel, 0.01s on Linux. Set `UseSharedCompilation=false` and `MSBUILDDISABLENODEREUSE=1` so no server is left running. **macOS x86_64 costs 4h22m41s of a 5h01m run** - against 1h27m54s for Linux arm64 - because that runner is roughly 2x slower than the M1 one at compiling and about 8x slower at spawning processes, which is what a tree of 70+ autotools packages does all day. curl's 576 configure probes take 21s on macos-14 and 178s there. It already runs with `continue-on-error`, so build it only once its published archive is older than `vars.MACOS_X86_MAX_AGE_DAYS` (default 7) rather than on every thirdparty change. `workflow_dispatch` with `force_build` still builds it unconditionally. **Nothing was cached between runs.** ccache is installed on the runners but nothing routed through it and nothing persisted it, so a thirdparty change that touches one package still rebuilt all ~80. Set `ENABLE_THIRDPARTY_CCACHE=ON` and carry `$CCACHE_DIR` across runs with `actions/cache`. The cache is saved even when the build fails, which is what makes the `max_attempts=3` retry in the prerelease job affordable - today a flaky download mirror can cost three full builds. This needs apache/doris to honour `ENABLE_THIRDPARTY_CCACHE`; until that lands it is an unused variable and the cache stays empty. `success` and `failure` now list every build job. `success` had to move to `!cancelled() && !failure()`, because a skipped `build-macos-x86_64` would otherwise skip it too and leave the release note stuck at BUILDING - which is the state that stops all future scheduled runs.
hello-stephen
approved these changes
Aug 17, 2026
This was referenced Aug 17, 2026
[chore](thirdparty) drop the hadoop-libs 3.3.6.6 build and use 3.4.2.4 everywhere
apache/doris#66837
Closed
morningman
added a commit
to morningman/doris
that referenced
this pull request
Aug 17, 2026
…nd cache ccache Two of the four fixes in apache/doris-thirdparty#410 apply to this workflow too. thrift's configure finds the dotnet the runner images ship, so `make` runs `dotnet build -c Release` for lib/netstd. The build itself finishes in seconds and then the compiler server it leaves behind holds the stdout it inherited for its full keep-alive while nothing happens. Between "Build succeeded" and "Making install in compiler/cpp" in apache/doris-thirdparty run 31988123966: 600.03s on macos-14, 598.79s on macos-15-intel, 0.01s on Linux. Both macOS jobs here pay it. `UseSharedCompilation=false` and `MSBUILDDISABLENODEREUSE=1` stop the server from being started. And nothing was cached between runs: ccache is installed on every runner but nothing routed compilation through it, so a change to one package recompiled all ~80, at `-j 2` on two of the three jobs. Turn on ENABLE_THIRDPARTY_CCACHE and carry the cache with the ccache-action this repository already vendors. max-size is 1G per job so that all three together stay inside what is left of the repository's 10 GB Actions cache budget next to the 5G BE-UT-macOS cache. Note that a cache saved by a pull request run is only visible to that same pull request, so this pays off across pushes to one branch; sharing it across pull requests would need this workflow to run on master. The other two fixes in that PR do not apply here: there is no Docker job to decouple, and no macOS x86_64 job to build less often.
morningman
added a commit
to morningman/doris
that referenced
this pull request
Aug 18, 2026
…nd cache ccache Two of the four fixes in apache/doris-thirdparty#410 apply to this workflow too. thrift's configure finds the dotnet the runner images ship, so `make` runs `dotnet build -c Release` for lib/netstd. The build itself finishes in seconds and then the compiler server it leaves behind holds the stdout it inherited for its full keep-alive while nothing happens. Between "Build succeeded" and "Making install in compiler/cpp" in apache/doris-thirdparty run 31988123966: 600.03s on macos-14, 598.79s on macos-15-intel, 0.01s on Linux. Both macOS jobs here pay it. `UseSharedCompilation=false` and `MSBUILDDISABLENODEREUSE=1` stop the server from being started. And nothing was cached between runs: ccache is installed on every runner but nothing routed compilation through it, so a change to one package recompiled all ~80, at `-j 2` on two of the three jobs. Turn on ENABLE_THIRDPARTY_CCACHE and carry the cache with the ccache-action this repository already vendors. max-size is 1G per job so that all three together stay inside what is left of the repository's 10 GB Actions cache budget next to the 5G BE-UT-macOS cache. Note that a cache saved by a pull request run is only visible to that same pull request, so this pays off across pushes to one branch; sharing it across pull requests would need this workflow to run on master. The other two fixes in that PR do not apply here: there is no Docker job to decouple, and no macOS x86_64 job to build less often.
morningman
added a commit
to apache/doris
that referenced
this pull request
Aug 19, 2026
… and slim it down, and add ccache (#66842) ### What problem does this PR solve? Issue Number: close #xxx Related PR: apache/doris-thirdparty#410 Problem Summary: Third-party build changes, all measured against [apache/doris-thirdparty run 31988123966](https://github.com/apache/doris-thirdparty/actions/runs/31988123966). Supersedes #66837 and #66839, which are now closed. **First half of a two-PR split.** This PR only changes what the third-party build produces (`thirdparty/**` and `.github/workflows/build-thirdparty.yml`). The consumer side - `cloud/`, `be/src/io/hdfs_builder.cpp`, `build.sh`, `run-be-ut.sh`, `run-cloud-ut.sh`, the thirdparty lifecycle test and the `env.sh` azure default - is in #66908, which has to land **after** this one and after the prebuilt archives have been rebuilt from it. See "Merge order" at the bottom. --- ## 1. Only build hadoop-libs 3.4.2.4 The tree carried two libhdfs builds: `build_hadoop_libs` built the **3.3.6.6** fork into `installed/{include,lib}/hadoop_hdfs/`, `build_hadoop_libs_3_4` built the **3.4.2.4** fork into `.../hadoop_hdfs_3_4/`, and on Linux both ran - 11m38s of the x86_64 build and 10m00s of the arm64 one, for a fork only the cloud module still consumed. The two prefixes were also mixed up on the consumer side: `be/src/io/fs/hdfs.h` and `be/CMakeLists.txt` were on 3.4.2.4, while `be/src/io/hdfs_builder.cpp` and all of `cloud/` were on 3.3.6.6. Both headers share the `LIBHDFS_HDFS_H` include guard, so `hdfs_builder.cpp` compiled against the 3.3.6.6 header while BE linked the 3.4.2.4 archive. Straightening that out is #66908. 3.4.2.4 stays where it is, under `hadoop_hdfs_3_4/`, so a build-env image from this change can still compile older Doris branches. The 3.3.6.6 build and its `hadoop_hdfs/` prefix are gone, and `hadoop_libs_3_4` now runs on Linux and macOS alike. ## 2. Build azure everywhere, and stop building what Doris does not link **It was only ever built on x86_64 Linux.** `env.sh` forced `DISABLE_BUILD_AZURE=ON` on aarch64 and macOS, and `build_azure` skipped `Darwin` a second time, so BE and the cloud meta-service lost `+AZURE_BLOB` and `+AZURE_STORAGE_VAULT` there. Nothing in the recipe was genuinely x86_64-Linux-only, just two details: `-ldl` fails on Apple (there is no libdl; those symbols are in libSystem), and vcpkg ships no prebuilt cmake/ninja/ curl for aarch64 Linux so it needs `VCPKG_FORCE_SYSTEM_BINARIES=1`. The triplet is now derived from `uname` rather than left to vcpkg's host detection, and `build_azure` no longer skips Darwin. `DISABLE_BUILD_AZURE` was answering two questions at once: whether the third-party tree *carries* azure, and whether BE and the cloud meta-service *link* it. The archive has to grow the libraries before anything can link them, so the two are split here - the third-party build reads `DISABLE_THIRDPARTY_BUILD_AZURE` and leaves `env.sh`'s switch to the consumers, whose default still skips azure on aarch64 and macOS until #66908. **It was the most expensive package in the tree** - 26m06s of the 2h43m x86_64 Linux build - almost none of it for Doris: | phase | time | |---|---| | `vcpkg install` | 22m11s | | - protobuf 5.29.3 | 12m29s | | - opentelemetry-cpp | 4m35s | | - abseil | 1m58s | | - openssl 1.1.1n | 1m38s | | - curl / libxml2 / zlib / uAMQP / utf8-range | ~1m30s | | azure SDK configure + build | ~3m55s | protobuf, abseil and utf8-range were pulled in **only** because `vcpkg.json` declared `opentelemetry-cpp` unconditionally - roughly 19 of the 26 minutes. Doris does not use azure's OpenTelemetry tracing. On top of that vcpkg builds every port twice (`Building x64-linux-dbg` then `-rel`) and Doris links only the release halves, and the SDK builds appconfiguration, attestation, eventhubs, keyvault, tables, template, uAMQP, storage-files-datalake, storage-files-shares and storage-queues, while `be/cmake/thirdparty.cmake` links exactly four targets: `azure-core`, `azure-identity`, `azure-storage-blobs`, `azure-storage-common`. So the patch drops `opentelemetry-cpp` and the uAMQP C libraries from `vcpkg.json`, flips `DISABLE_AMQP` / `DISABLE_AZURE_CORE_OPENTELEMETRY` to `ON` and trims the sub-projects to the four Doris links; `build_azure` writes an overlay triplet that sets `VCPKG_BUILD_TYPE release`. What is left for vcpkg to build is curl, libxml2, openssl 1.1.1n and zlib, release only. ## 3. ccache for the third-party build `ENABLE_THIRDPARTY_CCACHE=ON` (off by default) exports `CMAKE_C_COMPILER_LAUNCHER` / `CMAKE_CXX_COMPILER_LAUNCHER`, which CMake initialises from the environment, so no cmake invocation in the script changes. It deliberately does not prefix `CC`/`CXX`: CMake splits `"ccache clang"` into the compiler plus a `CMAKE_<LANG>_FLAGS` entry, and that leaks into whatever a package exports. Autotools packages are left alone. `.github/workflows/build-thirdparty.yml` turns it on and carries the cache with the `ccache-action` this repository already vendors. Today a change to one package recompiles all ~80, at `-j 2` on two of the three jobs. `max-size` is **1G** per job so the three together stay inside what is left of the repository's 10 GB Actions cache budget next to the 5G `BE-UT-macOS` cache. Note that a cache saved by a pull request run is only visible to that same pull request, so this pays off across pushes to one branch; sharing it across pull requests would need this workflow to run on master, the way `be-ut-mac.yml` uses a schedule for exactly that reason. Happy to drop this hunk if the cache budget is too tight. ## 4. Retry a failed vcpkg download instead of losing the azure build All three jobs of run 32032837067 built 73 of the 74 packages and then failed on the 74th, azure, in the same place: vcpkg downloading `github.com/madler/zlib` - HTTP 429 on Linux and macOS, 500 on macOS-arm64, while the Actions cache service was also answering "Our services aren't available right now". azure is the only package that fetches its own sources (everything else goes through `download-thirdparty.sh`, which has a mirror and a fallback), its own retry is three attempts inside one second, and it is the last package built - so each job threw away a finished tree over one file. And the exposure just tripled now that azure builds on all three platforms. So the configure is retried with a backoff long enough for a rate-limit window to pass (2, 4, 6, 8 minutes) and **only** when the failure was a download - a port that will not compile still stops on the first attempt. Ports that did build come back from vcpkg's binary cache, and `VCPKG_DOWNLOADS` now points outside the directory `build_azure` wipes on entry. `thirdparty/test/azure-vcpkg-retry-test.sh` drives `build_azure` against a stubbed cmake and covers both halves, in seconds, in the script-test job. ## Also: the macOS jobs stalled for 10 minutes each Carried over from apache/doris-thirdparty#410. thrift's configure finds the dotnet the runner images ship, so `make` runs `dotnet build -c Release` for `lib/netstd`. The build finishes in seconds and then the compiler server it leaves behind holds the stdout it inherited for its full keep-alive while nothing happens: | runner | "Build succeeded" -> "Making install in compiler/cpp" | |---|---| | macos-14 | **600.03s** | | macos-15-intel | **598.79s** | | ubuntu-22.04 | 0.01s | `UseSharedCompilation=false` and `MSBUILDDISABLENODEREUSE=1` stop the server from being started. Both macOS jobs in this workflow pay this today. The other two fixes in apache/doris-thirdparty#410 do not apply here: there is no Docker job to decouple, and no macOS x86_64 job to build less often. ## Merge order 1. **This PR.** Its own CI is unaffected by either half of the split: `env.sh` is untouched, so BE still links no azure on macOS, and `build.sh` still agrees with the lifecycle test on the `hadoop_hdfs/` sentinel. 2. The apache/doris-thirdparty automation rebuilds the prebuilt archives. It triggers on `^thirdparty/` and takes ~3h; only then do the darwin-arm64 and linux-aarch64 archives carry azure. The aarch64 build-env image has to be rebuilt too. 3. **#66908.** Between 1 and 3, a Linux tree that picks up a freshly built third-party archive will not find `installed/lib/hadoop_hdfs/native/libhdfs.a`, which is still what `build.sh`'s `LAST_THIRDPARTY_LIB` looks for, and will start a full third-party rebuild. Nothing in CI consumes such an archive for a BE or cloud build - `be-ut-mac.yml` is on Darwin, where the sentinel is `libbrotlienc.a`, and `build-extension.yml` only takes `thrift` out of the Linux archive - but that is the window #66908 closes. ### Release note None ### Check List (For Author) - Test - [x] Manual test (add detailed scripts or steps below) This PR touches `thirdparty/**`, so its own `Build Third Party Libraries` run builds the whole tree on ubuntu-22.04, macos-15 and macos-14 - which is exactly where the hadoop change, the azure slimming and, for the first time, azure on macOS arm64 get exercised. **aarch64 Linux is the one path this workflow does not cover**; that is only built in apache/doris-thirdparty. The azure patch was verified to apply cleanly to a pristine `azure-sdk-for-cpp-azure-core_1.16.0` tarball, and the resulting `vcpkg.json` parses and declares only `curl`, `libxml2`, `openssl`, `wil`. `actionlint` is clean on the workflow, and the four `thirdparty/test/*.sh` scripts the script-test job runs pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Run 31988123966
took 5h01m end to end. Per-job:
Four things fall out of that, and this PR addresses each.
1. The Docker image waited 1h39m for nothing
update-dockeronly consumesdoris-thirdparty-prebuilt-linux-x86_64.tar.xz,which the Linux job published at 05:19. But as
needs: buildit depended onthe whole matrix, so it did not start until 06:58 - held up by macOS x86_64,
which finished at 06:56.
A matrix job cannot be depended on per-leg, so the build job becomes a reusable
workflow (
.github/workflows/build-target.yml) called once per platform.update-dockernow depends onbuild-linux-x86_64alone. The provenance check italready runs against the archive is unchanged.
Splitting this way is also what makes items 3 and 4 below expressible per
platform. The package lists and the ldb-toolchain choice, which the matrix entries
used to carry, are derived from
RUNNER_OS/RUNNER_ARCHinside the reusableworkflow - the two Linux entries and the two macOS entries were identical except
for the toolchain script.
2. Every macOS build sat idle for exactly 10 minutes
thrift's configure finds the dotnet the runner images ship, so
makerunsdotnet build -c Releaseforlib/netstd. The build itself finishes in seconds(
Build succeeded, 12s on macos-14, 46s on macos-15-intel), and then the log goescompletely silent before
Making install in compiler/cpp:An exact 10 minutes on both Macs is not compute, it is the compiler server that
dotnet buildleaves behind holding the stdout it inherited for its fullkeep-alive.
UseSharedCompilation=falseandMSBUILDDISABLENODEREUSE=1stop itfrom being started at all. 10 minutes back on each macOS job.
3. macOS x86_64 is 87% of the pipeline's wall clock
macos-15-intelis roughly 2x slower than the M1 runner at compiling(lance-c 41m43s vs 22m19s, grpc 2.1x, arrow 2.3x) and about 8x slower at
spawning processes - which is what a tree of 70+ autotools packages does all day.
Identical configure scripts, same probe count:
It already runs with
continue-on-error, so it is now built only once itspublished archive is older than
vars.MACOS_X86_MAX_AGE_DAYS(default 7)rather than on every thirdparty change. The prerelease job reads the asset's
updatedAtoff the release, so this self-regulates with no extra cron.workflow_dispatchwithforce_buildstill builds it unconditionally, and thethreshold is a repository variable so it can be retuned without a code change.
4. Nothing was cached between runs
ccacheis installed on every runner, but nothing routed compilation through itand nothing persisted it, so a thirdparty change touching one package still
rebuilt all ~80 from scratch. This sets
ENABLE_THIRDPARTY_CCACHE=ONand carries$CCACHE_DIRacross runs withactions/cache(CCACHE_MAXSIZE=2Gcompressed,four platforms inside the repository's 10 GB Actions cache budget).
The cache is saved even when the build fails, which is the point: the
prerelease job retries a failed build up to
max_attempts=3, and today a flakydownload mirror can cost three consecutive four-hour builds.
This half only does something once apache/doris honours
ENABLE_THIRDPARTY_CCACHE(apache/doris#66839). Until then it is an unusedenvironment variable and the cache stays empty - the workflow is safe to merge in
either order.
Also
successandfailurenow list every build job.successhad to move to!cancelled() && !failure():build-macos-x86_64is skipped on most runs, and askipped dependency would otherwise skip
successtoo, leaving the release notestuck at
BUILDING- which is precisely the state that stops all future scheduledruns from doing anything.
Testing
actionlintis clean on both files. The behaviour itself can only be exercised byrunning the pipeline; the reusable workflow is a faithful move of the existing
steps, with
matrix.config.name/matrix.config.osreplaced byinputs.target/RUNNER_OS/RUNNER_ARCH, plus the ccache and dotnet environment above.Worth a
workflow_dispatchrun withforce_build=truebefore relying on it, sothat all four platforms and the Docker job are exercised in one go.