Skip to content

[chore](thirdparty) drop the hadoop-libs 3.3.6.6 build and use 3.4.2.4 everywhere - #66837

Closed
morningman wants to merge 1 commit into
apache:masterfrom
morningman:thirdparty-hadoop-only-3-4-2-4
Closed

[chore](thirdparty) drop the hadoop-libs 3.3.6.6 build and use 3.4.2.4 everywhere#66837
morningman wants to merge 1 commit into
apache:masterfrom
morningman:thirdparty-hadoop-only-3-4-2-4

Conversation

@morningman

@morningman morningman commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

The tree carried two libhdfs builds:

  • build_hadoop_libs built the 3.3.6.6 fork into installed/{include,lib}/hadoop_hdfs/
  • build_hadoop_libs_3_4 built the 3.4.2.4 fork into installed/{include,lib}/hadoop_hdfs_3_4/

On Linux both ran. In the latest scheduled apache/doris-thirdparty build that is
~10 extra minutes of every third-party build (11m38s for the 3.3.6.6 fork on the
x86_64 runner, 10m00s on the arm64 one) for a fork that only the cloud module
still consumed.

The two prefixes were also mixed up on the consumer side:

consumer header library
be/src/io/fs/hdfs.h hadoop_hdfs_3_4/hdfs.h (3.4.2.4)
be/src/io/hdfs_builder.cpp hadoop_hdfs/hdfs.h (3.3.6.6)
be/CMakeLists.txt hadoop_hdfs_3_4/native/libhdfs.a (3.4.2.4)
cloud/src/recycler/hdfs_accessor.{h,cpp} hadoop_hdfs/hdfs.h (3.3.6.6)
cloud/CMakeLists.txt hadoop_hdfs/native/libhdfs.a (3.3.6.6)

Both headers use the same LIBHDFS_HDFS_H include guard, so hdfs_builder.cpp
compiled against the 3.3.6.6 header (it came first) while BE linked the 3.4.2.4
archive.

This PR builds 3.4.2.4 only and moves the remaining 3.3.6.6 consumers onto it:

  • thirdparty/vars.sh: drop HADOOP_LIBS_* and its TP_ARCHIVES entry; the
    HADOOP_LIBS_3_4_* variables are untouched.
  • thirdparty/build-thirdparty.sh: drop build_hadoop_libs; hadoop_libs_3_4
    now runs on Linux and macOS alike, so the platform branch around it goes away.
    The install prefix stays hadoop_hdfs_3_4/ so a build-env image built from
    this change can still compile older Doris branches.
  • cloud/CMakeLists.txt and cloud/src/recycler/hdfs_accessor.{h,cpp}: 3.4.2.4.
  • be/src/io/hdfs_builder.cpp: drop the 3.3.6.6 include - io/fs/hdfs.h,
    included two lines below, already re-exports the 3.4.2.4 header.
  • build.sh (LAST_THIRDPARTY_LIB sentinel), run-be-ut.sh, run-cloud-ut.sh
    and thirdparty/test/arrow-paimon-lifecycle-test.sh follow the same prefix.
    The unit-test runners still stage the jars at lib/hadoop_hdfs/, so the
    classpath loops below them are unchanged.

Net effect: BE and the cloud meta-service compile and link against the same
libhdfs, and every third-party build drops one full Hadoop native build.

Release note

None

Check List (For Author)

  • Test

    • No need to test or manual test. Explain why:
      • Other reason: this only changes which Hadoop fork the third-party
        build produces and which prefix each consumer reads. A wrong path fails
        the compile or the link outright, so BE/cloud compilation plus the
        existing HDFS regression cases cover it.
  • Behavior changed:

    • Yes. installed/lib/hadoop_hdfs/ and installed/include/hadoop_hdfs/
      no longer exist in the prebuilt third-party archive - only hadoop_hdfs_3_4/
      remains. The third-party prebuilt has to be rebuilt before this lands in a
      build-env image, and anything outside this repository that reads the
      hadoop_hdfs/ prefix needs updating.
  • Does this need documentation?

    • No.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

…4 everywhere

The tree carried two libhdfs builds. `build_hadoop_libs` built the 3.3.6.6
fork into `installed/{include,lib}/hadoop_hdfs/`, `build_hadoop_libs_3_4`
built the 3.4.2.4 fork into `.../hadoop_hdfs_3_4/`, and on Linux both ran -
roughly 10 extra minutes of every third-party build for a fork only the cloud
module still consumed.

The two prefixes were also mixed up on the consumer side. BE linked
`hadoop_hdfs_3_4/native/libhdfs.a`, but `be/src/io/hdfs_builder.cpp` included
`hadoop_hdfs/hdfs.h` - the 3.3.6.6 header - while `be/src/io/fs/hdfs.h`
included the 3.4.2.4 one. Both headers share the LIBHDFS_HDFS_H include guard,
so that translation unit compiled against 3.3.6.6 and linked 3.4.2.4. The cloud
meta-service used 3.3.6.6 for both.

Build 3.4.2.4 only, and move the remaining 3.3.6.6 consumers onto it:

- thirdparty: drop `build_hadoop_libs` and the `HADOOP_LIBS_*` variables;
  `hadoop_libs_3_4` now runs on Linux and macOS alike. The install prefix stays
  `hadoop_hdfs_3_4/` so a build-env image from this change can still compile
  older Doris branches.
- cloud: link `hadoop_hdfs_3_4/native/libhdfs.a` and include the matching header.
- `be/src/io/hdfs_builder.cpp`: drop the 3.3.6.6 include - `io/fs/hdfs.h`, two
  lines below, already re-exports the 3.4.2.4 one.
- `build.sh`, `run-be-ut.sh`, `run-cloud-ut.sh` and the thirdparty lifecycle
  test follow the same prefix.
@morningman
morningman force-pushed the thirdparty-hadoop-only-3-4-2-4 branch from 7dd6d97 to fd5e56c Compare August 17, 2026 09:02
@morningman morningman changed the title [chore](thirdparty) build only hadoop-libs 3.4.2.4 and drop the 3.3.6.6 fork [chore](thirdparty) drop the hadoop-libs 3.3.6.6 build and use 3.4.2.4 everywhere Aug 17, 2026
@morningman

Copy link
Copy Markdown
Contributor Author

Superseded by #66842, which combines this with the other third-party change and additionally applies the two applicable fixes from apache/doris-thirdparty#410 to .github/workflows/build-thirdparty.yml.

@morningman morningman closed this Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants