Skip to content

[chore](thirdparty) build hadoop_libs_3_4 on macOS too - #66800

Closed
morningman wants to merge 1 commit into
apache:masterfrom
morningman:build-hadoop-libs-3-4-on-macos
Closed

[chore](thirdparty) build hadoop_libs_3_4 on macOS too#66800
morningman wants to merge 1 commit into
apache:masterfrom
morningman:build-hadoop-libs-3-4-on-macos

Conversation

@morningman

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Related PR: #66783

Problem Summary:

build-thirdparty.sh adds the hadoop packages to its default package list under uname -s == Linux alone, so no macOS build has ever installed lib/hadoop_hdfs_3_4/ — neither a developer's local prefix, nor the doris-thirdparty-prebuilt-darwin-x86_64 / -arm64 archives, which apache/doris-thirdparty's automation produces by running this very script with no package arguments. A macOS prefix carries libhdfs3 (lib64/libhdfs3.a, include/hdfs/) and nothing else.

That was correct while hadoop's native build could not complete on a Mac at all. apache/doris-thirdparty#407 fixed it, and hadoop-3.4.2.3-for-doris — selected since #66783 — carries the fix, so the 3.4 fork builds here now.

hadoop_libs, the 3.3.6 fork, stays Linux-only: it has none of those fixes, and nothing built on macOS reads its hadoop_hdfs/ prefix — the BE includes hadoop_hdfs_3_4/hdfs.h, and the cloud module, the only consumer of the 3.3.6 one, is not built on macOS.

Making the BE link hadoop's libhdfs.a on macOS instead of libhdfs3 (be/CMakeLists.txt still selects libhdfs3 there) is a separate change. It needs this one first, and it needs the published prebuilt archives rebuilt with it — which the doris-thirdparty automation does on the next thirdparty/ change, i.e. on this one.

Release note

None

Check List (For Author)

  • Test

    • Manual test (add detailed scripts or steps below)

    Verified on macOS 26.5 / arm64. With this change the default package list ends with hadoop_libs_3_4 and still contains no hadoop_libs; HADOOP_LIBS_3_4 is already unconditional in TP_ARCHIVES, so the source is downloaded on macOS today and download-thirdparty.sh needs no change. The package build itself was verified in [chore](thirdparty) upgrade HADOOP_LIBS_3_4 to hadoop-3.4.2.3-for-doris and build macOS CI on JDK 17 #66783: build-thirdparty.sh hadoop_libs_3_4 exits 0 and installs

    installed/lib/hadoop_hdfs_3_4/native/  libhdfs.a         1.4M  arm64
                                           libhadoop.dylib   160K  arm64
                                           libhdfs.dylib     135K  arm64
                                           libhdfspp.dylib   8.9M  arm64
    installed/include/hadoop_hdfs_3_4/     hdfs.h
    installed/lib/hadoop_hdfs_3_4/         common/ (5 jars)  hdfs/ (10 jars)
    
  • Behavior changed:

    • Yes. A macOS third-party build now also builds hadoop 3.4.2.3 and installs lib/hadoop_hdfs_3_4/. Linux is byte for byte unchanged.
  • Does this need documentation?

    • No.

build-thirdparty.sh adds the hadoop packages to its default package list
under `uname -s == Linux` alone, so no macOS build has ever installed
lib/hadoop_hdfs_3_4 - neither a developer's local prefix, nor the
doris-thirdparty-prebuilt-darwin-{x86_64,arm64} archives, which
apache/doris-thirdparty's automation produces by running this very script
with no package arguments. A macOS prefix carries libhdfs3 and nothing else.

That was correct while hadoop's native build could not complete on a Mac at
all. apache/doris-thirdparty#407 fixed it and hadoop-3.4.2.3-for-doris,
selected since apache#66783, carries the fix, so the 3.4 fork builds here now.

hadoop_libs, the 3.3.6 fork, stays Linux-only: it has none of those fixes,
and nothing built on macOS reads its hadoop_hdfs/ prefix - the BE includes
hadoop_hdfs_3_4/hdfs.h, and the cloud module, the only consumer of the 3.3.6
one, is not built on macOS.

Making the BE link hadoop's libhdfs.a there instead of libhdfs3 is a separate
change: it needs this one first, and needs the published prebuilt archives
rebuilt with it.

Verified on macOS 26.5 / arm64: the default package list now ends with
hadoop_libs_3_4 and still contains no hadoop_libs, and HADOOP_LIBS_3_4 is
already unconditional in TP_ARCHIVES, so the source is downloaded here today
and download-thirdparty.sh needs no change. The package build itself was
verified in apache#66783 - build-thirdparty.sh hadoop_libs_3_4 exits 0 and installs
libhdfs.a plus libhadoop/libhdfs/libhdfspp dylibs, hdfs.h and the jars.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants