[fix](iceberg) Backport external write hardening to branch-4.1

[Gabriel39]

What problem does this PR solve?

This is a semantic backport of the external-write and schema-evolution fixes from #66345, #66348, #66408, #66529, and #66750 to branch-4.1.

The source PRs were implemented on the connector-plugin architecture, while branch-4.1 still uses the legacy external-table stack. This change ports the invariants and failure handling to the branch-specific owners instead of mechanically cherry-picking plugin classes.

Key behavior preserved on branch-4.1:

• Keep Iceberg planning, serialization, and commit on one metadata generation; add exact snapshot/branch overwrite OCC checks and retry-state reset.
• Handle nested schema evolution safely, including typed initial defaults, required-field history, alias collisions, nested partition sources, stable nested IDs, and collection parent null masks.
• Transfer external-file ownership only through the final bounded report, retain rollback ownership until FE acknowledgement, and keep rolling compatibility for non-Iceberg report vectors.
• Bound asynchronous writer and multi-partition sorter memory, including queue admission ownership and largest-sorter revocation.
• Preserve writer close errors and file sizes; harden Iceberg data/delete-file cleanup and Hive deferred multipart validation/rollback.
• Isolate Azure multipart writers with UUID-namespaced block IDs and commit the exact staged IDs.
• Make orphan-file deletion retained-snapshot-aware, prefix-safe, GC-enabled, and bounded.
• Preserve nested comments as SQL literals and isolate MTMV test job lookup.

The Paimon validation-detail fix from #66408 and canonical Paimon capitalization from #66750 are already present in the legacy Paimon paths on branch-4.1; their observable contracts were verified and no plugin-only code was introduced.

Architecture-specific notes

• The Iceberg scan compatibility gate evaluates historical schema/default/equality-delete requirements only while a smooth-upgrade source BE exists. Equality deletes are still checked across every delete manifest in the selected snapshot.
• Empty overwrite scans the retained target branch rather than the main branch, so divergent branch files cannot be mixed into the delete set.
• The legacy Hive planner explicitly advertises deferred Azure multipart support, matching the connector-planner handshake from the source change.

Tests

• FE focused reactor suite: 174 tests, 0 failures, 0 errors, 1 skipped.
• Target-branch overwrite regression verified red/green against a divergent main branch.
• Regression framework SuiteJobLookupTest: 1 test passed.
• All 35 changed C++ translation units present in the ASAN compile database built successfully.
• Azure BE source passed a syntax-only compile with the repository C++ flags (it is excluded from the local build configuration).
• clang-format 16 check passed for all 63 changed C/C++ source and header files.
• FE Checkstyle passed with 0 violations.

The full local BE unit-test link is unavailable because the configured third-party Arrow bundle lacks a header required by an unchanged source file; targeted compilation covers every changed translation unit.

Pick from master: #66345 #66348 #66408 #66529 #66750

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[Gabriel39]

run buildall

[Gabriel39]

/review

[hello-stephen]

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.48% (1909/2464)
Line Coverage	64.30% (34172/53141)
Region Coverage	64.44% (17299/26845)
Branch Coverage	53.98% (9267/17166)

[github-actions]

Request changes: this review found 11 actionable correctness and rolling-compatibility defects.

The blocking paths group into four areas: external-file report ownership can delete committed Iceberg files or strand Hive multipart uploads; mixed-version Hive/Iceberg writes can fail after staging or silently use the wrong partition source; Scanner V2 loses or misinterprets Iceberg requiredness and initial defaults; and current nested non-identity merge routing constructs transforms from the wrong type.

Critical-checkpoint summary: reader lifecycle and collection parent-null handling were cleared; async reservation transfer, spill accounting, and dependency wakeups were cleared; overwrite OCC/branch behavior and the orphan-file action remained fail-closed; but schema mapping/default materialization, external compatibility, nested partition routing, and report ownership have the inline blockers below. No additional user-provided focus was supplied. The review converged in Round 2 after both full-scope passes and the separate risk challenge returned no new valuable findings. Per the review workflow, no builds or source modifications were performed.

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Request changes: this review found seven additional actionable defects.

Critical-checkpoint summary:

• Correctness and data safety: top-level non-finite Iceberg defaults fail in FE before the new BE parser is reachable; orphan deletion can race a concurrent commit; and rejected final reports can leave prepared Paimon files without an abort owner.
• Resource lifetime and admission: cold partition fan-out and the final spill merger can both allocate far beyond their successful reservations.
• External compatibility and performance: a positive equality-delete summary still causes unnecessary remote manifest traversal during rolling-upgrade checks.
• Tests: the expanded complex-evolution suite has stale and missing golden results, so its checked-in expectations cannot pass.

The final-report state machine for the other inspected Iceberg/Hive/MaxCompute paths, nested partition routing, object-store publication, branch overwrite/OCC, reader lifecycle, and collection null handling were cleared. All 115 changed paths were reviewed, all 11 existing inline threads were duplicate-fenced, and two additional concerns were dismissed as pre-existing after authoritative-preimage checks. No additional user-provided focus was supplied. Three full rounds converged unanimously with no new valuable findings in the final round. Per the review instructions, no builds or source modifications were performed.

[hello-stephen]

FE UT Coverage Report

Increment line coverage 31.84% (213/669) 🎉
Increment coverage report
Complete coverage report

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Request changes: seven additional, distinct defects remain (five P1, two P2).

Findings:

1. [P1] Failed rewrite actions can permanently retain the new Iceberg commit fence.
2. [P1] Target-file rollover can invoke a full sort without reserving destination/permutation workspace.
3. [P1] Nontrivial ARRAY/MAP rematerialization still allocates an entry-sized parent mask when no descendant consumes it.
4. [P1] Paimon backend close failures are discarded after FE has accepted commit payloads.
5. [P1] Cold-writer reserve estimation copies a potentially huge first value before memory admission.
6. [P2] The V2 gate rejects selected-snapshot/ref scans based on schemas that cannot contribute reachable files.
7. [P2] The V2 gate expands a nested-pruned root to unrequested descendants, so an unused sibling can reject a V1-safe scan.

Review checkpoints:

• Architecture and layer boundaries: traced Scanner/TableReader/mapper/file-reader responsibilities and FE-to-Thrift-to-BE ownership. No separate boundary inversion survived review; the schema-gate and materialization defects are called out inline.
• Reader and transaction lifecycle/contracts: checked split/EOF/error cleanup, async queue tokens, report ACK/retry/finalizers, Iceberg fence/OCC paths, and writer close/rollback. The rewrite-fence leak and post-ACK Paimon close regression remain blocking.
• Schema mapping/materialization: checked stable field IDs, recursive defaults/optionality/binary markers, nested projection, ARRAY/MAP offsets/null maps, selected schemas, and partition paths. Three projection/materialization defects remain inline.
• Filtering, deletes, and pushdown: traced predicates plus equality/position-delete paths and row/schema alignment; no additional nonduplicate issue survived beyond prior live threads and the V2 projection gates above.
• Format boundaries and external compatibility: checked Iceberg V1/V2 rolling upgrade, time travel/refs, external schema evolution, Hive/Paimon reporting, nested partitions, and Azure/S3 multipart identity. The two mixed-version false rejections are inline; other reproduced compatibility concerns are already threaded.
• Performance and observability: reviewed admission, sorter rollover/spill/final merge, hot-path collection masks, report-size handling, counters, and cleanup visibility. Three unreserved/unbounded allocation paths remain inline.
• Tests: reviewed all changed BE/FE/regression tests. Coverage does not exercise the seven reported witnesses; no builds or tests were run, as required by the review prompt.
• User focus: no additional user-provided focus was supplied.

Status: capped/incomplete after the maximum three rounds because the final normal and final risk passes both produced new distinct findings; this is not a convergence claim.

[hello-stephen]

FE UT Coverage Report

Increment line coverage 33.73% (252/747) 🎉
Increment coverage report
Complete coverage report

[Gabriel39]

run buildall

[Gabriel39]

/review

[hello-stephen]

FE UT Coverage Report

Increment line coverage 33.26% (287/863) 🎉
Increment coverage report
Complete coverage report