[chore](thirdparty) build azure on every platform and stop building what Doris does not link - #66839
Closed
morningman wants to merge 1 commit into
Closed
[chore](thirdparty) build azure on every platform and stop building what Doris does not link#66839morningman wants to merge 1 commit into
morningman wants to merge 1 commit into
Conversation
…hat Doris does not link Two things about azure-sdk-for-cpp. It only ever got built on x86_64 Linux. env.sh forced DISABLE_BUILD_AZURE=ON on aarch64 and macOS, and build_azure skipped Darwin a second time, so BE and the cloud meta-service lost +AZURE_BLOB and +AZURE_STORAGE_VAULT on those platforms. Nothing in the recipe was actually x86_64-Linux-specific except two details: `-ldl`, which fails on Apple because there is no libdl there, and vcpkg needing VCPKG_FORCE_SYSTEM_BINARIES on aarch64 Linux, where it ships no prebuilt tools. Both are handled now, and the triplet is derived from uname instead of being left to vcpkg's host detection. And it was by far the most expensive package in the tree - 26m06s of the 2h43m x86_64 Linux third-party build in apache/doris-thirdparty run 31988123966 - almost none of it for Doris's benefit: - 22m of the 26m was `vcpkg install`, and 19m of that was opentelemetry-cpp (4m35s) plus the protobuf 5.29.3 (12m29s), abseil (1m58s) and utf8-range it drags in. Doris does not use azure's OpenTelemetry tracing at all; the dependency was in vcpkg.json unconditionally. - vcpkg builds every port twice, debug and release, and Doris links only the release halves. - The SDK builds appconfiguration, attestation, eventhubs, keyvault, tables, template, uAMQP, datalake, file shares and queues. Doris links azure-core, azure-identity, azure-storage-common and azure-storage-blobs. So the patch drops opentelemetry-cpp and the uAMQP C libraries from vcpkg.json, flips DISABLE_AMQP and DISABLE_AZURE_CORE_OPENTELEMETRY to ON, and trims the sub-projects to the four Doris links; build_azure adds an overlay triplet that sets VCPKG_BUILD_TYPE=release. What is left to build is curl, libxml2, openssl 1.1.1n and zlib, release only. Also add an opt-in ENABLE_THIRDPARTY_CCACHE. A third-party build is cold every time, so a warm ccache turns a rebuild triggered by one changed package into minutes rather than hours. It exports CMAKE_<LANG>_COMPILER_LAUNCHER, which CMake picks up from the environment, rather than prefixing CC/CXX - that would put the compiler name into CMAKE_<LANG>_FLAGS. Autotools packages are left alone, and it is off unless asked for.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
Superseded by #66842, which combines this with the other third-party change and additionally applies the two applicable fixes from apache/doris-thirdparty#410 to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Two problems with
azure-sdk-for-cpp.1. It was only ever built on x86_64 Linux.
env.shforcedDISABLE_BUILD_AZURE=ONon aarch64 and macOS, andbuild_azureskipped
Darwina second time. As a result BE and the cloud meta-service lost+AZURE_BLOBand+AZURE_STORAGE_VAULTon ARM and macOS. Nothing in the recipewas genuinely x86_64-Linux-specific, only two details:
-ldl: Apple has no libdl at all (dlopen and friends live in libSystem), so-ldlfails the link there. It is now passed on non-Apple platforms only, bothin
build_azureand in the openssl vcpkg wrapper the existing patch touches.VCPKG_FORCE_SYSTEM_BINARIES=1there.The vcpkg triplet is now derived from
uname(x64-linux,arm64-linux,x64-osx,arm64-osx) instead of being left to vcpkg's host detection.2. It was the most expensive package in the tree, almost none of it for Doris.
In
apache/doris-thirdpartyrun 31988123966, azure took 26m06s of the 2h43mx86_64 Linux third-party build. Where it went:
vcpkg installprotobuf, abseil and utf8-range are there only because
vcpkg.jsondeclaresopentelemetry-cppunconditionally - roughly 19 of the 26 minutes. Doris doesnot use azure's OpenTelemetry tracing.
On top of that, vcpkg builds every port twice (
Building x64-linux-dbgthenBuilding x64-linux-rel) and Doris links only the release halves, and the SDKbuilds appconfiguration, attestation, eventhubs, keyvault, tables, template,
uAMQP, storage-files-datalake, storage-files-shares and storage-queues while
be/cmake/thirdparty.cmakelinks exactly four targets:azure-core,azure-identity,azure-storage-blobs,azure-storage-common.So this PR:
thirdparty/patches/azure-sdk-for-cpp-azure-core_1.16.0.patchto dropopentelemetry-cppand the uAMQP C libraries fromvcpkg.json, flipDISABLE_AMQP/DISABLE_AZURE_CORE_OPENTELEMETRYtoON, and trim thesub-project list to core + identity + storage-{common,blobs};
build_azurethat setsVCPKG_BUILD_TYPE release,which is the only supported way to ask vcpkg for a release-only build.
What is left for vcpkg to build is curl, libxml2, openssl 1.1.1n and zlib,
release only.
Also in this PR: an opt-in ccache for the third-party build.
ENABLE_THIRDPARTY_CCACHE=ONexportsCMAKE_C_COMPILER_LAUNCHER/CMAKE_CXX_COMPILER_LAUNCHER, which CMake initialises from the environment, so nocmake invocation in the script changes. It deliberately does not prefix
CC/CXXwith ccache: CMake would split
"ccache clang"into the compiler plus aCMAKE_<LANG>_FLAGSentry, and that leaks into whatever a package exports.Autotools packages are left alone. It is off by default because prefixing the
compiler changes how every package configures itself;
apache/doris-thirdpartyCI turns it on, where a warm ccache turns a rebuild triggered by one changed
package into minutes instead of hours.
Release note
Azure Blob Storage support (
+AZURE_BLOB,+AZURE_STORAGE_VAULT) is now built onaarch64 and macOS, not just x86_64 Linux.
Check List (For Author)
Test
Manual test (add detailed scripts or steps below)
The patch was verified to apply cleanly to a pristine
azure-sdk-for-cpp-azure-core_1.16.0tarball, and the resultingvcpkg.jsonparses and declares only
curl,libxml2,openssl,wil. The builditself is exercised by the
apache/doris-thirdpartypipeline, which is theonly place all four platforms are built.
Behavior changed:
prebuilt has to be rebuilt before this lands: BE/cloud on those platforms
will pass
-DBUILD_AZURE=ONand fail to link against an older prebuiltarchive that has no azure libraries.
DISABLE_BUILD_AZURE=ONstill opts out.installed/no longer contains the azure sub-libraries Doris does not link(appconfiguration, attestation, eventhubs, keyvault, tables, template,
storage-files-datalake, storage-files-shares, storage-queues), nor the
opentelemetry/protobuf/abseil artifacts vcpkg used to install under the
azure build tree.
Does this need documentation?