[spark] Externalize oversized input partition metadata

[JustinBinber]

Part of #10021.

Purpose

Spark currently embeds every Paimon Split in an InputPartition. For scans with many file descriptors, the serialized task payload can become large enough to put pressure on driver memory, RPC, retries, and speculation.

This PR adds an opt-in transport path for oversized input partition metadata. Small partitions keep the existing inline behavior. Large partitions are written to a shared seekable FileIO container, while Spark tasks carry only a compact descriptor and read their own byte range on the executor.

Changes

• Add session options for the external metadata path and inline threshold. The feature is disabled unless a path is configured; the default threshold is 128 MiB.
• Add a versioned, framed container format with per-split lengths, CRC32 validation, and footer statistics.
• Share one metadata container per scan and preserve bucket information in the external descriptor.
• Read external metadata through range reads on executors.
• Keep row count, split count, and estimated data bytes in the descriptor so metrics do not need to decode the payload.
• Clean up staged metadata on planning failure and at Spark application shutdown.
• Document the configuration and its trust requirements.

Scope

This PR only bounds the metadata carried by Spark tasks. It does not yet make driver-side split planning streaming, stream split decoding on executors, or split one exceptionally large DataSplit. Those are intentionally left as separate follow-up changes discussed in #10021.

Validation

• Spark 3 / JDK 8: focused unit tests and full SparkReadITCase
• Spark 4.1.2 / Scala 2.13 / JDK 17: the same test matrix
• Spotless, Checkstyle, Apache RAT, and git diff --check

A metadata-only probe with 350 DataSplits and 1,000 file descriptors per split reduced the serialized InputPartition payload from about 136.5 MB to a small descriptor. The full metadata remains in the external container; this result demonstrates task-payload reduction, not an end-to-end scan speedup.

[JingsongLi]

Requirement fit: PIVOT (triage: NO-GO for this PR as scoped)

Issue #10021 reports Driver OOM or restart around split discovery, task construction, and serialization. This PR only shrinks the serialized Spark InputPartition sent to tasks. PaimonBatch still starts with fully materialized inputPartitions and each partition's Seq[Split]; the externalized partition retains plannedSplits on the Driver; and the Executor still decodes the full split list. The 136.5 MB probe measures task payload rather than a completed scan. The integration test forces the path with a 1-byte threshold on a tiny table, so it does not show that a workload which currently fails can finish.

For this roughly 1,700-line change, the new shared staging path, serialization format, extra encoding pass, and file/broadcast cleanup lifecycle need a demonstrated end-to-end capacity benefit. Please provide a reproducible SQL workload that completes planning but fails specifically at task transport, then show it completes with this option enabled, including peak Driver/Executor memory and runtime at a representative file count. Alternatively, pivot to the planning or decode boundary that blocks the reported workload. I am closing this PR for now; it can be reconsidered with that evidence and a narrower implementation.