From 2bcb315ec6dd2811f9b9f4519223a9c412ecb09a Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 16 Aug 2026 22:08:14 +0800 Subject: [PATCH 1/2] [chore](thirdparty) upgrade HADOOP_LIBS_3_4 to hadoop-3.4.2.4-for-doris Picks up apache/doris-thirdparty#408: macOS no longer builds libhdfspp. That library does not link there. It is compiled against Homebrew LLVM's libc++ headers, configured with no Apple availability annotations, so std::exception_ptr's post-macOS-14 entry points are emitted whatever -mmacosx-version-min says, while -lc++ still resolves against the SDK's system libc++. Against a macOS 14 SDK that leaves __cxa_init_primary_exception and std::exception_ptr::__from_native_exception_pointer undefined, and the failing dylib takes the whole make down with it - libhdfs.a included. Doris never links libhdfspp (be/CMakeLists.txt sets BUILD_LIBHDFSPP OFF), so the fork now stops after the x-platform object libraries libhdfs pulls out of that tree. libhdfs.a, libhdfs.dylib, libhadoop.dylib, hdfs.h and the jars are unchanged, and Linux is unaffected. Co-Authored-By: Claude Opus 5 (1M context) --- thirdparty/CHANGELOG.md | 4 ++++ thirdparty/vars.sh | 8 ++++---- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/thirdparty/CHANGELOG.md b/thirdparty/CHANGELOG.md index 1cb2e96a53b8c8..8c049ea51bb3a7 100644 --- a/thirdparty/CHANGELOG.md +++ b/thirdparty/CHANGELOG.md @@ -2,6 +2,10 @@ This file contains version of the third-party dependency libraries in the build-env image. The docker build-env image is apache/doris, and the tag is `build-env-${version}` +## 20260816 + +- Modified: hadoop-libs 3.4.2.3 -> 3.4.2.4 + ## 20260814 - Modified: hadoop-libs 3.4.2.2 -> 3.4.2.3 diff --git a/thirdparty/vars.sh b/thirdparty/vars.sh index 211954c27a87c2..6a42bd00cf7963 100644 --- a/thirdparty/vars.sh +++ b/thirdparty/vars.sh @@ -504,10 +504,10 @@ HADOOP_LIBS_SOURCE="doris-thirdparty-hadoop-3.3.6.6-for-doris" HADOOP_LIBS_MD5SUM="13b66d5f2abffd1740e692b65df5962e" # libhdfs 3.4 -HADOOP_LIBS_3_4_DOWNLOAD="https://github.com/apache/doris-thirdparty/archive/refs/tags/hadoop-3.4.2.3-for-doris.tar.gz" -HADOOP_LIBS_3_4_NAME="hadoop-3.4.2.3-for-doris.tar.gz" -HADOOP_LIBS_3_4_SOURCE="doris-thirdparty-hadoop-3.4.2.3-for-doris" -HADOOP_LIBS_3_4_MD5SUM="a751a42f3a927c4a272a99d4e3788648" +HADOOP_LIBS_3_4_DOWNLOAD="https://github.com/apache/doris-thirdparty/archive/refs/tags/hadoop-3.4.2.4-for-doris.tar.gz" +HADOOP_LIBS_3_4_NAME="hadoop-3.4.2.4-for-doris.tar.gz" +HADOOP_LIBS_3_4_SOURCE="doris-thirdparty-hadoop-3.4.2.4-for-doris" +HADOOP_LIBS_3_4_MD5SUM="729a7ef9126f65255d1da124a88ffd50" # AvxToNeon AVX2NEON_DOWNLOAD="https://github.com/kunpengcompute/AvxToNeon/archive/refs/tags/v1.0.0.tar.gz" From 2c1fd8f3a7e2630f06eafb72545a60302000b8b7 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 16 Aug 2026 10:30:44 +0800 Subject: [PATCH 2/2] [chore](thirdparty) build hadoop_libs_3_4 on macOS too build-thirdparty.sh adds the hadoop packages to its default package list under `uname -s == Linux` alone, so no macOS build has ever installed lib/hadoop_hdfs_3_4 - neither a developer's local prefix, nor the doris-thirdparty-prebuilt-darwin-{x86_64,arm64} archives, which apache/doris-thirdparty's automation produces by running this very script with no package arguments. A macOS prefix carries libhdfs3 and nothing else. That was correct while hadoop's native build could not complete on a Mac at all. apache/doris-thirdparty#407 fixed it and hadoop-3.4.2.3-for-doris, selected since #66783, carries the fix, so the 3.4 fork builds here now. hadoop_libs, the 3.3.6 fork, stays Linux-only: it has none of those fixes, and nothing built on macOS reads its hadoop_hdfs/ prefix - the BE includes hadoop_hdfs_3_4/hdfs.h, and the cloud module, the only consumer of the 3.3.6 one, is not built on macOS. Making the BE link hadoop's libhdfs.a there instead of libhdfs3 is a separate change: it needs this one first, and needs the published prebuilt archives rebuilt with it. Verified on macOS 26.5 / arm64: the default package list now ends with hadoop_libs_3_4 and still contains no hadoop_libs, and HADOOP_LIBS_3_4 is already unconditional in TP_ARCHIVES, so the source is downloaded here today and download-thirdparty.sh needs no change. The package build itself was verified in #66783 - build-thirdparty.sh hadoop_libs_3_4 exits 0 and installs libhdfs.a plus libhadoop/libhdfs/libhdfspp dylibs, hdfs.h and the jars. Co-Authored-By: Claude Opus 5 (1M context) --- thirdparty/build-thirdparty.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh index f44122ca1f3f88..6f0676e565a33f 100755 --- a/thirdparty/build-thirdparty.sh +++ b/thirdparty/build-thirdparty.sh @@ -2372,6 +2372,10 @@ if [[ "${#packages[@]}" -eq 0 ]]; then ) if [[ "$(uname -s)" == 'Darwin' ]]; then read -r -a packages <<<"binutils gettext ${packages[*]}" + # hadoop_libs, the 3.3.6 fork, stays Linux-only: it carries none of the macOS fixes + # apache/doris-thirdparty#407 made to the 3.4 fork, and nothing built here reads its + # hadoop_hdfs/ prefix - the cloud module, its only other consumer, is Linux-only too. + read -r -a packages <<<"${packages[*]} hadoop_libs_3_4" elif [[ "$(uname -s)" == 'Linux' ]]; then read -r -a packages <<<"${packages[*]} hadoop_libs" read -r -a packages <<<"${packages[*]} hadoop_libs_3_4"