[chore](thirdparty) build only hadoop 3.4.2.4, build azure everywhere and slim it down, and add ccache

[morningman]

What problem does this PR solve?

Issue Number: close #xxx

Related PR: apache/doris-thirdparty#410

Problem Summary:

Three related third-party build changes, all measured against
apache/doris-thirdparty run 31988123966.
Supersedes #66837 and #66839, which are now closed.

---

1. Only build hadoop-libs 3.4.2.4

The tree carried two libhdfs builds: build_hadoop_libs built the 3.3.6.6 fork
into installed/{include,lib}/hadoop_hdfs/, build_hadoop_libs_3_4 built the
3.4.2.4 fork into .../hadoop_hdfs_3_4/, and on Linux both ran - 11m38s of the
x86_64 build and 10m00s of the arm64 one, for a fork only the cloud module still
consumed.

The two prefixes were also mixed up on the consumer side:

consumer	header	library
be/src/io/fs/hdfs.h	hadoop_hdfs_3_4/hdfs.h (3.4.2.4)	—
be/src/io/hdfs_builder.cpp	hadoop_hdfs/hdfs.h (3.3.6.6)	—
be/CMakeLists.txt	—	hadoop_hdfs_3_4/native/libhdfs.a (3.4.2.4)
cloud/src/recycler/hdfs_accessor.{h,cpp}	hadoop_hdfs/hdfs.h (3.3.6.6)	—
cloud/CMakeLists.txt	—	hadoop_hdfs/native/libhdfs.a (3.3.6.6)

Both headers share the LIBHDFS_HDFS_H include guard, so hdfs_builder.cpp
compiled against the 3.3.6.6 header (it came first) while BE linked the 3.4.2.4
archive.

3.4.2.4 stays where it is, under hadoop_hdfs_3_4/, so a build-env image from this
change can still compile older Doris branches. The 3.3.6.6 build and its
hadoop_hdfs/ prefix are gone, and cloud/, build.sh's LAST_THIRDPARTY_LIB
sentinel, run-be-ut.sh, run-cloud-ut.sh and the thirdparty lifecycle test move
to hadoop_hdfs_3_4/. The unit-test runners still stage the jars at
lib/hadoop_hdfs/, so the classpath loops below them are unchanged.

2. Build azure everywhere, and stop building what Doris does not link

It was only ever built on x86_64 Linux. env.sh forced
DISABLE_BUILD_AZURE=ON on aarch64 and macOS, and build_azure skipped Darwin a
second time, so BE and the cloud meta-service lost +AZURE_BLOB and
+AZURE_STORAGE_VAULT there. Nothing in the recipe was genuinely x86_64-Linux-only,
just two details: -ldl fails on Apple (there is no libdl; those symbols are in
libSystem), and vcpkg ships no prebuilt cmake/ninja/curl for aarch64 Linux so it
needs VCPKG_FORCE_SYSTEM_BINARIES=1. The triplet is now derived from uname
rather than left to vcpkg's host detection.

It was the most expensive package in the tree - 26m06s of the 2h43m x86_64 Linux
build - almost none of it for Doris:

phase	time
vcpkg install	22m11s
- protobuf 5.29.3	12m29s
- opentelemetry-cpp	4m35s
- abseil	1m58s
- openssl 1.1.1n	1m38s
- curl / libxml2 / zlib / uAMQP / utf8-range	~1m30s
azure SDK configure + build	~3m55s

protobuf, abseil and utf8-range were pulled in only because vcpkg.json declared
opentelemetry-cpp unconditionally - roughly 19 of the 26 minutes. Doris does not
use azure's OpenTelemetry tracing. On top of that vcpkg builds every port twice
(Building x64-linux-dbg then -rel) and Doris links only the release halves, and
the SDK builds appconfiguration, attestation, eventhubs, keyvault, tables, template,
uAMQP, storage-files-datalake, storage-files-shares and storage-queues, while
be/cmake/thirdparty.cmake links exactly four targets: azure-core,
azure-identity, azure-storage-blobs, azure-storage-common.

So the patch drops opentelemetry-cpp and the uAMQP C libraries from vcpkg.json,
flips DISABLE_AMQP / DISABLE_AZURE_CORE_OPENTELEMETRY to ON and trims the
sub-projects to the four Doris links; build_azure writes an overlay triplet that
sets VCPKG_BUILD_TYPE release. What is left for vcpkg to build is curl, libxml2,
openssl 1.1.1n and zlib, release only.

3. ccache for the third-party build

ENABLE_THIRDPARTY_CCACHE=ON (off by default) exports
CMAKE_C_COMPILER_LAUNCHER / CMAKE_CXX_COMPILER_LAUNCHER, which CMake initialises
from the environment, so no cmake invocation in the script changes. It deliberately
does not prefix CC/CXX: CMake splits "ccache clang" into the compiler plus a
CMAKE_<LANG>_FLAGS entry, and that leaks into whatever a package exports.
Autotools packages are left alone.

.github/workflows/build-thirdparty.yml turns it on and carries the cache with the
ccache-action this repository already vendors. Today a change to one package
recompiles all ~80, at -j 2 on two of the three jobs.

max-size is 1G per job so the three together stay inside what is left of the
repository's 10 GB Actions cache budget next to the 5G BE-UT-macOS cache. Note
that a cache saved by a pull request run is only visible to that same pull request,
so this pays off across pushes to one branch; sharing it across pull requests would
need this workflow to run on master, the way be-ut-mac.yml uses a schedule for
exactly that reason. Happy to drop this hunk if the cache budget is too tight.

Also: the macOS jobs stalled for 10 minutes each

Carried over from apache/doris-thirdparty#410. thrift's configure finds the dotnet
the runner images ship, so make runs dotnet build -c Release for lib/netstd.
The build finishes in seconds and then the compiler server it leaves behind holds
the stdout it inherited for its full keep-alive while nothing happens:

runner	"Build succeeded" -> "Making install in compiler/cpp"
macos-14	600.03s
macos-15-intel	598.79s
ubuntu-22.04	0.01s

UseSharedCompilation=false and MSBUILDDISABLENODEREUSE=1 stop the server from
being started. Both macOS jobs in this workflow pay this today.

The other two fixes in apache/doris-thirdparty#410 do not apply here: there is no
Docker job to decouple, and no macOS x86_64 job to build less often.

Release note

Azure Blob Storage support (+AZURE_BLOB, +AZURE_STORAGE_VAULT) is now built on
aarch64 and macOS, not just x86_64 Linux.

Check List (For Author)

• Test

  • Manual test (add detailed scripts or steps below)

    This PR touches thirdparty/** and env.sh, so its own
    Build Third Party Libraries run builds the whole tree on ubuntu-22.04,
    macos-15 and macos-14 - which is exactly where the hadoop change, the azure
    slimming and, for the first time, azure on macOS arm64 get exercised.
    aarch64 Linux is the one path this workflow does not cover; that is only
    built in apache/doris-thirdparty.

    The azure patch was verified to apply cleanly to a pristine
    azure-sdk-for-cpp-azure-core_1.16.0 tarball, and the resulting vcpkg.json
    parses and declares only curl, libxml2, openssl, wil. actionlint is
    clean on the workflow.

• Behavior changed:

  • Yes.
    • installed/{include,lib}/hadoop_hdfs/ no longer exists in the prebuilt
      third-party archive; only hadoop_hdfs_3_4/ remains. Anything outside this
      repository reading the hadoop_hdfs/ prefix needs updating.
    • Azure is now built and linked on aarch64 and macOS. The third-party
      prebuilt has to be rebuilt before this lands: BE/cloud on those platforms
      will pass -DBUILD_AZURE=ON and fail to link against an older prebuilt
      archive that has no azure libraries. DISABLE_BUILD_AZURE=ON still opts out.
    • installed/ no longer contains the azure sub-libraries Doris does not link.

• Does this need documentation?

  • No.

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[github-actions]

PR approved by at least one committer and no changes requested.

[github-actions]

PR approved by anyone and no changes requested.

[morningman]

run compile