[feature](be) Add file scanner v2 readers by Gabriel39 · Pull Request #65046 · apache/doris

Gabriel39 · 2026-06-30T13:08:08Z

What problem does this PR solve?

Problem Summary: Add the file scanner v2 reader stack for external file scans, including native readers for Parquet, CSV/TEXT, JSON, JNI-backed table readers, schema projection, column mapping, predicate handling, reader statistics, page cache support, and related BE/FE integration. This also restores affected Parquet LZO regression cases by adding Doris thirdparty Arrow LZO page decompression support for file scanner v2.
Support file scanner v2 readers for external file scan paths, including LZO-compressed Parquet reads in the new Parquet reader path.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-06-30T13:08:13Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Gabriel39 · 2026-06-30T13:09:07Z

run buildall

hello-stephen · 2026-06-30T13:48:37Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.39% (1896/2450)
Line Coverage	64.44% (34063/52862)
Region Coverage	64.84% (17528/27032)
Branch Coverage	54.02% (9394/17390)

Gabriel39 · 2026-06-30T14:05:44Z

run buildall

hello-stephen · 2026-06-30T15:30:05Z

FE UT Coverage Report

Increment line coverage 60.00% (3/5) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-06-30T21:22:37Z

BE Regression && UT Coverage Report

Increment line coverage 84.17% (13996/16628) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.58% (29571/39652)
Line Coverage	58.53% (323352/552450)
Region Coverage	55.34% (271359/490351)
Branch Coverage	56.60% (119468/211082)

hello-stephen · 2026-06-30T21:29:29Z

FE Regression Coverage Report

Increment line coverage 40.00% (2/5) 🎉
Increment coverage report
Complete coverage report

Gabriel39 · 2026-07-01T01:21:32Z

/review

github-actions

Automated review for PR 65046. I found four issues that should be addressed before merge: a FileScannerV2 runtime-filter rewrite correctness bug, a default-on rollout mismatch for enable_file_scanner_v2, a removed timezone regression assertion, and generated output whitespace that fails git diff --check.

### What problem does this PR solve? Issue Number: close #xxx Related PR: #63893 Problem Summary: Add the file scanner v2 reader stack for external file scans, including native readers for Parquet, CSV/TEXT, JSON, JNI-backed table readers, schema projection, column mapping, predicate handling, reader statistics, page cache support, and related BE/FE integration. This also restores affected Parquet LZO regression cases by adding Doris thirdparty Arrow LZO page decompression support for file scanner v2. The change keeps VDirectInPredicate source-compatible with existing ordinary two-argument construction by defaulting the new HybridSet child-type flag to true. Dictionary-code rewrites can still pass false explicitly, while existing runtime filter tests continue to compile with the old call shape. Review follow-up fixes make RuntimeFilterExpr global-index slot rewriting update the executable _impl tree, document enable_file_scanner_v2 as default-on to match the FE default, and trim generated regression outputs so diff hygiene passes. ### Release note Support file scanner v2 readers for external file scan paths, including LZO-compressed Parquet reads in the new Parquet reader path. ### Check List (For Author) - Test: Manual test - Verified apache-arrow-17.0.0-lzo.patch applies with patch -p1 --dry-run against Arrow 17 column_reader.cc - Ran bash -n thirdparty/build-thirdparty.sh thirdparty/download-thirdparty.sh - Ran build-support/clang-format.sh - Ran git diff --check - Attempted ./run-be-ut.sh --run --filter='RuntimeFilterExprSamplingTest.deep_clone_clones_impl_tree'; local sandboxed run could not complete because the BE UT script required JDK 17 setup first, then needed submodule metadata writes and GitHub access for thirdparty dependencies. Escalated retry was not approved before timeout. - Attempted ./run-be-ut.sh --run --filter='FileScannerV2Test.RewriteSlotRefsToGlobalIndexMatrix'; local sandboxed run could not complete because the BE UT script needed submodule metadata writes and GitHub access for thirdparty dependencies. Escalated retry was not approved before timeout. - Full BE unit tests and external regression tests were not run in this local environment - Behavior changed: Yes. Adds file scanner v2 reader behavior and enables LZO-compressed Parquet reads through the new reader path - Does this need documentation: No

Gabriel39 · 2026-07-01T02:12:34Z

run buildall

Gabriel39 · 2026-07-01T02:12:42Z

/review

github-actions

Review complete. I did not find a new in-scope, non-duplicate issue to add beyond the existing inline review threads.

Scope and duplicate handling: I checked the provided review prompt, repo review skill, required AGENTS files, existing review threads/comments, focus file, changed-file list, and shared ledger before submitting. The already-raised scanner V2 runtime-filter rewrite, scanner default/docs, timezone-test, and whitespace comments were treated as duplicate-suppression context. One initially suspicious local diff in StmtExecutor was dismissed because it is absent from the authoritative live PR file list and GitHub rejected the path as unresolved.

Critical checkpoints: scanner V2 selection/session propagation, FE format planning, runtime-filter rewrite handling, aggregate/count pushdown, file-reader paths, changed tests/outputs, and style hygiene were reviewed statically. git diff --check 9d7d3a2c4f437e0c2962bde1a49848ed0535063c..HEAD is clean. I did not run builds or test suites because this worktree is not initialized and thirdparty/installed is missing in the runner.

Subagent conclusions: optimizer-rewrite and tests-session-config both completed the final convergence round on the corrected no-inline-comment set with NO_NEW_VALUABLE_FINDINGS.

yiguolei · 2026-07-01T02:34:59Z

+    int64_t conversion_failure_null_map_offset = 0;
+};
+
+inline bool decoded_column_view_row_is_null(const DecodedColumnView& view, int64_t row) {


这里为什么不把DecodedColumnView ，变成一个class？然后底下这堆方法是这个class的member？

yiguolei · 2026-07-01T02:47:01Z

               std::dynamic_pointer_cast<VSlotRef>(get_child(0)) != nullptr;
    }

+    Status clone_node(VExprSPtr* cloned_expr) const override {


这个clone node 是干啥用的？

hello-stephen · 2026-07-01T02:48:43Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.39% (1896/2450)
Line Coverage	64.50% (34097/52862)
Region Coverage	64.91% (17546/27032)
Branch Coverage	54.07% (9402/17390)

yiguolei · 2026-07-01T07:00:41Z

+    return slot_info.is_file_slot;
+}
+
+Status rewrite_slot_refs_to_global_index(


这个方法需要加注释，并且注释里得有例子

这个runtime filter 会不会被共享，所以我们修改他的结构是有风险的？

yiguolei · 2026-07-01T07:02:01Z

+Status FileScannerV2::_build_table_conjuncts(VExprContextSPtrs* conjuncts) const {
+    DORIS_CHECK(conjuncts != nullptr);
+    conjuncts->clear();
+    conjuncts->reserve(_conjuncts.size());


这里为啥拿到conjuncts 之后要rewrite 一下？为啥rewrite 是在每个scanner 都rewrite 一次，而不是file scan operator rewrite 一次?

github-actions · 2026-07-01T07:13:13Z

PR approved by at least one committer and no changes requested.

Gabriel39 requested review from BiteTheDDDDt, airborne12, csun5285, eldenmoon, gavinchou, liaoxin01, morningman, yiguolei and zclllyybb as code owners June 30, 2026 13:08

Gabriel39 force-pushed the refact_reader_branch branch from 6fa6952 to 47c9df5 Compare June 30, 2026 14:04

github-actions Bot requested changes Jul 1, 2026

View reviewed changes

Comment thread be/src/exec/scan/file_scanner_v2.cpp

Comment thread be/src/exec/operator/file_scan_operator.cpp

Comment thread regression-test/suites/external_table_p0/hive/test_hive_date_timezone.groovy

Comment thread regression-test/suites/external_table_p0/tvf/test_hdfs_parquet_group4.groovy

Gabriel39 force-pushed the refact_reader_branch branch from 47c9df5 to 8fe2181 Compare July 1, 2026 02:02

github-actions Bot reviewed Jul 1, 2026

View reviewed changes

yiguolei reviewed Jul 1, 2026

View reviewed changes

yiguolei approved these changes Jul 1, 2026

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jul 1, 2026

Uh oh!

Conversation

Gabriel39 commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jun 30, 2026

Uh oh!

Gabriel39 commented Jun 30, 2026

Uh oh!

hello-stephen commented Jun 30, 2026

Cloud UT Coverage Report

Uh oh!

Gabriel39 commented Jun 30, 2026

Uh oh!

hello-stephen commented Jun 30, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jun 30, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jun 30, 2026

FE Regression Coverage Report

Uh oh!

Gabriel39 commented Jul 1, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Gabriel39 commented Jul 1, 2026

Uh oh!

Gabriel39 commented Jul 1, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

yiguolei Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

yiguolei Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Jul 1, 2026

Cloud UT Coverage Report

Uh oh!

yiguolei Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

yiguolei Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Gabriel39 commented Jun 30, 2026 •

edited

Loading