From 131c39b0418e3bdb79b7562629208c3578d8721b Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 16 Aug 2026 10:30:44 +0800 Subject: [PATCH] [chore](thirdparty) build hadoop_libs_3_4 on macOS too build-thirdparty.sh adds the hadoop packages to its default package list under `uname -s == Linux` alone, so no macOS build has ever installed lib/hadoop_hdfs_3_4 - neither a developer's local prefix, nor the doris-thirdparty-prebuilt-darwin-{x86_64,arm64} archives, which apache/doris-thirdparty's automation produces by running this very script with no package arguments. A macOS prefix carries libhdfs3 and nothing else. That was correct while hadoop's native build could not complete on a Mac at all. apache/doris-thirdparty#407 fixed it and hadoop-3.4.2.3-for-doris, selected since #66783, carries the fix, so the 3.4 fork builds here now. hadoop_libs, the 3.3.6 fork, stays Linux-only: it has none of those fixes, and nothing built on macOS reads its hadoop_hdfs/ prefix - the BE includes hadoop_hdfs_3_4/hdfs.h, and the cloud module, the only consumer of the 3.3.6 one, is not built on macOS. Making the BE link hadoop's libhdfs.a there instead of libhdfs3 is a separate change: it needs this one first, and needs the published prebuilt archives rebuilt with it. Verified on macOS 26.5 / arm64: the default package list now ends with hadoop_libs_3_4 and still contains no hadoop_libs, and HADOOP_LIBS_3_4 is already unconditional in TP_ARCHIVES, so the source is downloaded here today and download-thirdparty.sh needs no change. The package build itself was verified in #66783 - build-thirdparty.sh hadoop_libs_3_4 exits 0 and installs libhdfs.a plus libhadoop/libhdfs/libhdfspp dylibs, hdfs.h and the jars. Co-Authored-By: Claude Opus 5 (1M context) --- thirdparty/build-thirdparty.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh index f44122ca1f3f88..6f0676e565a33f 100755 --- a/thirdparty/build-thirdparty.sh +++ b/thirdparty/build-thirdparty.sh @@ -2372,6 +2372,10 @@ if [[ "${#packages[@]}" -eq 0 ]]; then ) if [[ "$(uname -s)" == 'Darwin' ]]; then read -r -a packages <<<"binutils gettext ${packages[*]}" + # hadoop_libs, the 3.3.6 fork, stays Linux-only: it carries none of the macOS fixes + # apache/doris-thirdparty#407 made to the 3.4 fork, and nothing built here reads its + # hadoop_hdfs/ prefix - the cloud module, its only other consumer, is Linux-only too. + read -r -a packages <<<"${packages[*]} hadoop_libs_3_4" elif [[ "$(uname -s)" == 'Linux' ]]; then read -r -a packages <<<"${packages[*]} hadoop_libs" read -r -a packages <<<"${packages[*]} hadoop_libs_3_4"