[chore](thirdparty) drop the hadoop-libs 3.3.6.6 build and use 3.4.2.4 everywhere - #66837
Closed
morningman wants to merge 1 commit into
Closed
[chore](thirdparty) drop the hadoop-libs 3.3.6.6 build and use 3.4.2.4 everywhere#66837morningman wants to merge 1 commit into
morningman wants to merge 1 commit into
Conversation
morningman
requested review from
Gabriel39,
gavinchou and
liaoxin01
as code owners
August 17, 2026 08:55
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
…4 everywhere
The tree carried two libhdfs builds. `build_hadoop_libs` built the 3.3.6.6
fork into `installed/{include,lib}/hadoop_hdfs/`, `build_hadoop_libs_3_4`
built the 3.4.2.4 fork into `.../hadoop_hdfs_3_4/`, and on Linux both ran -
roughly 10 extra minutes of every third-party build for a fork only the cloud
module still consumed.
The two prefixes were also mixed up on the consumer side. BE linked
`hadoop_hdfs_3_4/native/libhdfs.a`, but `be/src/io/hdfs_builder.cpp` included
`hadoop_hdfs/hdfs.h` - the 3.3.6.6 header - while `be/src/io/fs/hdfs.h`
included the 3.4.2.4 one. Both headers share the LIBHDFS_HDFS_H include guard,
so that translation unit compiled against 3.3.6.6 and linked 3.4.2.4. The cloud
meta-service used 3.3.6.6 for both.
Build 3.4.2.4 only, and move the remaining 3.3.6.6 consumers onto it:
- thirdparty: drop `build_hadoop_libs` and the `HADOOP_LIBS_*` variables;
`hadoop_libs_3_4` now runs on Linux and macOS alike. The install prefix stays
`hadoop_hdfs_3_4/` so a build-env image from this change can still compile
older Doris branches.
- cloud: link `hadoop_hdfs_3_4/native/libhdfs.a` and include the matching header.
- `be/src/io/hdfs_builder.cpp`: drop the 3.3.6.6 include - `io/fs/hdfs.h`, two
lines below, already re-exports the 3.4.2.4 one.
- `build.sh`, `run-be-ut.sh`, `run-cloud-ut.sh` and the thirdparty lifecycle
test follow the same prefix.
morningman
force-pushed
the
thirdparty-hadoop-only-3-4-2-4
branch
from
August 17, 2026 09:02
7dd6d97 to
fd5e56c
Compare
3 tasks
Contributor
Author
|
Superseded by #66842, which combines this with the other third-party change and additionally applies the two applicable fixes from apache/doris-thirdparty#410 to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
The tree carried two libhdfs builds:
build_hadoop_libsbuilt the 3.3.6.6 fork intoinstalled/{include,lib}/hadoop_hdfs/build_hadoop_libs_3_4built the 3.4.2.4 fork intoinstalled/{include,lib}/hadoop_hdfs_3_4/On Linux both ran. In the latest scheduled
apache/doris-thirdpartybuild that is~10 extra minutes of every third-party build (11m38s for the 3.3.6.6 fork on the
x86_64 runner, 10m00s on the arm64 one) for a fork that only the cloud module
still consumed.
The two prefixes were also mixed up on the consumer side:
be/src/io/fs/hdfs.hhadoop_hdfs_3_4/hdfs.h(3.4.2.4)be/src/io/hdfs_builder.cpphadoop_hdfs/hdfs.h(3.3.6.6)be/CMakeLists.txthadoop_hdfs_3_4/native/libhdfs.a(3.4.2.4)cloud/src/recycler/hdfs_accessor.{h,cpp}hadoop_hdfs/hdfs.h(3.3.6.6)cloud/CMakeLists.txthadoop_hdfs/native/libhdfs.a(3.3.6.6)Both headers use the same
LIBHDFS_HDFS_Hinclude guard, sohdfs_builder.cppcompiled against the 3.3.6.6 header (it came first) while BE linked the 3.4.2.4
archive.
This PR builds 3.4.2.4 only and moves the remaining 3.3.6.6 consumers onto it:
thirdparty/vars.sh: dropHADOOP_LIBS_*and itsTP_ARCHIVESentry; theHADOOP_LIBS_3_4_*variables are untouched.thirdparty/build-thirdparty.sh: dropbuild_hadoop_libs;hadoop_libs_3_4now runs on Linux and macOS alike, so the platform branch around it goes away.
The install prefix stays
hadoop_hdfs_3_4/so a build-env image built fromthis change can still compile older Doris branches.
cloud/CMakeLists.txtandcloud/src/recycler/hdfs_accessor.{h,cpp}: 3.4.2.4.be/src/io/hdfs_builder.cpp: drop the 3.3.6.6 include -io/fs/hdfs.h,included two lines below, already re-exports the 3.4.2.4 header.
build.sh(LAST_THIRDPARTY_LIBsentinel),run-be-ut.sh,run-cloud-ut.shand
thirdparty/test/arrow-paimon-lifecycle-test.shfollow the same prefix.The unit-test runners still stage the jars at
lib/hadoop_hdfs/, so theclasspath loops below them are unchanged.
Net effect: BE and the cloud meta-service compile and link against the same
libhdfs, and every third-party build drops one full Hadoop native build.
Release note
None
Check List (For Author)
Test
build produces and which prefix each consumer reads. A wrong path fails
the compile or the link outright, so BE/cloud compilation plus the
existing HDFS regression cases cover it.
Behavior changed:
installed/lib/hadoop_hdfs/andinstalled/include/hadoop_hdfs/no longer exist in the prebuilt third-party archive - only
hadoop_hdfs_3_4/remains. The third-party prebuilt has to be rebuilt before this lands in a
build-env image, and anything outside this repository that reads the
hadoop_hdfs/prefix needs updating.Does this need documentation?