[chore](thirdparty) build hadoop_libs_3_4 on macOS too - #66800
Closed
morningman wants to merge 1 commit into
Closed
Conversation
build-thirdparty.sh adds the hadoop packages to its default package list
under `uname -s == Linux` alone, so no macOS build has ever installed
lib/hadoop_hdfs_3_4 - neither a developer's local prefix, nor the
doris-thirdparty-prebuilt-darwin-{x86_64,arm64} archives, which
apache/doris-thirdparty's automation produces by running this very script
with no package arguments. A macOS prefix carries libhdfs3 and nothing else.
That was correct while hadoop's native build could not complete on a Mac at
all. apache/doris-thirdparty#407 fixed it and hadoop-3.4.2.3-for-doris,
selected since apache#66783, carries the fix, so the 3.4 fork builds here now.
hadoop_libs, the 3.3.6 fork, stays Linux-only: it has none of those fixes,
and nothing built on macOS reads its hadoop_hdfs/ prefix - the BE includes
hadoop_hdfs_3_4/hdfs.h, and the cloud module, the only consumer of the 3.3.6
one, is not built on macOS.
Making the BE link hadoop's libhdfs.a there instead of libhdfs3 is a separate
change: it needs this one first, and needs the published prebuilt archives
rebuilt with it.
Verified on macOS 26.5 / arm64: the default package list now ends with
hadoop_libs_3_4 and still contains no hadoop_libs, and HADOOP_LIBS_3_4 is
already unconditional in TP_ARCHIVES, so the source is downloaded here today
and download-thirdparty.sh needs no change. The package build itself was
verified in apache#66783 - build-thirdparty.sh hadoop_libs_3_4 exits 0 and installs
libhdfs.a plus libhadoop/libhdfs/libhdfspp dylibs, hdfs.h and the jars.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
This was referenced Aug 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related PR: #66783
Problem Summary:
build-thirdparty.shadds the hadoop packages to its default package list underuname -s == Linuxalone, so no macOS build has ever installedlib/hadoop_hdfs_3_4/— neither a developer's local prefix, nor thedoris-thirdparty-prebuilt-darwin-x86_64/-arm64archives, which apache/doris-thirdparty's automation produces by running this very script with no package arguments. A macOS prefix carries libhdfs3 (lib64/libhdfs3.a,include/hdfs/) and nothing else.That was correct while hadoop's native build could not complete on a Mac at all. apache/doris-thirdparty#407 fixed it, and
hadoop-3.4.2.3-for-doris— selected since #66783 — carries the fix, so the 3.4 fork builds here now.hadoop_libs, the 3.3.6 fork, stays Linux-only: it has none of those fixes, and nothing built on macOS reads itshadoop_hdfs/prefix — the BE includeshadoop_hdfs_3_4/hdfs.h, and the cloud module, the only consumer of the 3.3.6 one, is not built on macOS.Making the BE link hadoop's
libhdfs.aon macOS instead of libhdfs3 (be/CMakeLists.txtstill selects libhdfs3 there) is a separate change. It needs this one first, and it needs the published prebuilt archives rebuilt with it — which the doris-thirdparty automation does on the nextthirdparty/change, i.e. on this one.Release note
None
Check List (For Author)
Test
Verified on macOS 26.5 / arm64. With this change the default package list ends with
hadoop_libs_3_4and still contains nohadoop_libs;HADOOP_LIBS_3_4is already unconditional inTP_ARCHIVES, so the source is downloaded on macOS today anddownload-thirdparty.shneeds no change. The package build itself was verified in [chore](thirdparty) upgrade HADOOP_LIBS_3_4 to hadoop-3.4.2.3-for-doris and build macOS CI on JDK 17 #66783:build-thirdparty.sh hadoop_libs_3_4exits 0 and installsBehavior changed:
lib/hadoop_hdfs_3_4/. Linux is byte for byte unchanged.Does this need documentation?