branch-4.1: [feat] add Parquet metadata TVF #58972 by xylaaaaa · Pull Request #61357 · apache/doris

xylaaaaa · 2026-03-16T03:10:55Z

Cherry-pick #58972 to branch-4.1

What problem does this PR solve?

Related PR: [feat] add Parquet metadata TVF #58972

Add Parquet metadata TVF.

Cherry-pick commit

96798c6520e - [feat] add Parquet metadata TVF ([feat] add Parquet metadata TVF #58972)

Thearas · 2026-03-16T03:11:00Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

xylaaaaa · 2026-03-16T03:12:37Z

run buildall

Copilot

Pull request overview

This PR introduces a new Parquet metadata table-valued function (TVF) pipeline (parquet_meta + convenience aliases) that lets users query Parquet footer/schema/statistics/bloom info via the existing metadata scan framework (FE thrift params → BE MetaScanner reader), along with regression + unit tests.

Changes:

Add TMetadataType.PARQUET and a new thrift payload TParquetMetadataParams, then wire a new BE ParquetMetadataReader into MetaScanner.
Add FE catalog + Nereids bindings for parquet_meta and aliases (parquet_file_metadata, parquet_kv_metadata, parquet_bloom_probe) and parameter validation / glob expansion.
Add regression coverage (S3/HDFS/local) and BE unit tests for shared Parquet utility helpers.

Reviewed changes

Copilot reviewed 18 out of 22 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
regression-test/suites/external_table_p0/tvf/test_parquet_meta_tvf.groovy	New regression suite covering parquet_meta modes across S3/HDFS/local + error cases
regression-test/data/external_table_p0/tvf/test_parquet_meta_tvf.out	Golden output for the new regression suite
regression-test/data/external_table_p0/tvf/meta.parquet	Test Parquet file used by regression suite
regression-test/data/external_table_p0/tvf/kvmeta.parquet	Test Parquet file containing KV metadata
regression-test/data/external_table_p0/tvf/empty.parquet	Empty Parquet test file
regression-test/data/external_table_p0/tvf/bloommeta.parquet	Parquet file with bloom filter metadata for probe mode
gensrc/thrift/Types.thrift	Add `TMetadataType.PARQUET`
gensrc/thrift/PlanNodes.thrift	Add `TParquetMetadataParams` and attach to `TMetaScanRange`
fe/fe-core/src/main/java/org/apache/doris/tablefunction/TableValuedFunctionIf.java	Register parquet_meta + alias function names in FE TVF factory
fe/fe-core/src/main/java/org/apache/doris/tablefunction/ParquetMetadataTableValuedFunction.java	Implement FE-side parquet_meta TVF param validation + glob expansion + scan-range building
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/visitor/TableValuedFunctionVisitor.java	Add Nereids visitor hook for ParquetMeta
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/ParquetMeta.java	Nereids table function binding for `parquet_meta`
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/ParquetFileMetadata.java	Nereids binding for `parquet_file_metadata` alias
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/ParquetKvMetadata.java	Nereids binding for `parquet_kv_metadata` alias
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/ParquetBloomProbe.java	Nereids binding for `parquet_bloom_probe` alias
fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinTableValuedFunctions.java	Register new Parquet TVFs as built-ins for Nereids
be/test/vec/exec/format/parquet/parquet_utils_test.cpp	Unit tests for new parquet_utils helpers
be/src/vec/exec/scan/meta_scanner.cpp	Route `TMetadataType::PARQUET` to the new BE reader
be/src/vec/exec/format/table/parquet_utils.h	Shared constants + helper APIs for Parquet metadata formatting
be/src/vec/exec/format/table/parquet_utils.cpp	Implement helper routines (type strings, stats decode, path mapping, etc.)
be/src/vec/exec/format/table/parquet_metadata_reader.h	New BE GenericReader that reads Parquet footers and emits rows per mode
be/src/vec/exec/format/table/parquet_metadata_reader.cpp	Implementation of the Parquet metadata scan (schema/metadata/file/kv/bloom modes)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...ain/java/org/apache/doris/nereids/trees/expressions/functions/table/ParquetFileMetadata.java

+    protected TableValuedFunctionIf toCatalogFunction() {
+        try {
+            Map<String, String> arguments = getTVFProperties().getMap();
+            arguments.put("mode", "parquet_file_metadata");
+            return new ParquetMetadataTableValuedFunction(arguments);


.../main/java/org/apache/doris/nereids/trees/expressions/functions/table/ParquetKvMetadata.java

+    protected TableValuedFunctionIf toCatalogFunction() {
+        try {
+            Map<String, String> arguments = getTVFProperties().getMap();
+            arguments.put("mode", "parquet_kv_metadata");
+            return new ParquetMetadataTableValuedFunction(arguments);


.../main/java/org/apache/doris/nereids/trees/expressions/functions/table/ParquetBloomProbe.java

+    protected TableValuedFunctionIf toCatalogFunction() {
+        try {
+            Map<String, String> arguments = getTVFProperties().getMap();
+            arguments.put("mode", "parquet_bloom_probe");
+            return new ParquetMetadataTableValuedFunction(arguments);


fe/fe-core/src/main/java/org/apache/doris/tablefunction/ParquetMetadataTableValuedFunction.java

+        String scheme = null;
+        try {
+            scheme = new URI(parsedPath).getScheme();
+        } catch (URISyntaxException e) {
+            scheme = null;


be/src/vec/exec/format/table/parquet_metadata_reader.cpp

+            case tparquet::Type::INT32: {
+                int64_t v = std::stoll(literal);
+                int32_t v32 = static_cast<int32_t>(v);
+                out->assign(reinterpret_cast<const char*>(&v32), sizeof(int32_t));
+                return Status::OK();


be/src/vec/exec/format/table/parquet_metadata_reader.h

+namespace doris::vectorized {
+class Block;
+
+// Lightweight reader that surfaces Parquet footer metadata as a table-valued scan.
+// It reads only file footers (no data pages) and emits either schema rows or


be/src/vec/exec/format/table/parquet_metadata_reader.cpp

+    } else if (params.__isset.paths && !params.paths.empty()) {
+        resolved_paths.assign(params.paths.begin(), params.paths.end());
+    } else {
+        return Status::InvalidArgument("Property 'path' must be set for parquet_meta");


be/src/vec/exec/format/table/parquet_metadata_reader.cpp

+namespace doris::vectorized {
+
+using namespace parquet_utils;
+
+class ParquetMetadataReader::ModeHandler {


hello-stephen · 2026-03-16T03:38:18Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.15% (1788/2259)
Line Coverage	64.53% (31962/49530)
Region Coverage	65.33% (15986/24468)
Branch Coverage	55.95% (8512/15214)

hello-stephen · 2026-03-16T04:22:59Z

FE UT Coverage Report

Increment line coverage 2.39% (5/209) 🎉
Increment coverage report
Complete coverage report

xylaaaaa · 2026-03-16T05:13:43Z

run buildall

doris-robot · 2026-03-16T05:43:58Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.15% (1788/2259)
Line Coverage	64.47% (31933/49530)
Region Coverage	65.31% (15979/24468)
Branch Coverage	55.88% (8502/15214)

yiguolei · 2026-03-16T13:01:24Z

run buildall

hello-stephen · 2026-03-16T14:08:11Z

FE UT Coverage Report

Increment line coverage 2.39% (5/209) 🎉
Increment coverage report
Complete coverage report

### What problem does this PR solve? Expose Parquet file metadata via a table-valued function for inspection and debugging. ### Issue Number: close #xxx ### Related PR: #xxx ### Problem Summary: - Add a Parquet metadata TVF so users can query Parquet file metadata via SQL. - Backend adds a Parquet metadata reader and scan path; frontend wires the TVF definition. - Enables easy inspection of partitions/row groups/column stats to aid troubleshooting.

…kets (apache#60938) ## Summary - adjust `test_parquet_meta_tvf` S3-mode checks to compare only stable columns - avoid asserting `file_name` / full S3 URI fields that vary by pipeline bucket - update the corresponding `.out` baseline for the changed query projections ## Why Different CI pipelines may use different bucket names, which causes false failures when full URI/file name columns are compared. ## Test - attempted: `./run-regression-test.sh --run -f external_table_p0/tvf/test_parquet_meta_tvf -forceGenOut` - in this environment it failed with S3 `FORBIDDEN` while reading regression parquet files

morningman · 2026-03-16T17:01:15Z

run buildall

hello-stephen · 2026-03-16T17:54:48Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.15% (1788/2259)
Line Coverage	64.50% (31947/49530)
Region Coverage	65.35% (15989/24468)
Branch Coverage	55.88% (8501/15214)

xylaaaaa · 2026-03-17T05:12:05Z

run buildall

doris-robot · 2026-03-17T08:56:35Z

BE UT Coverage Report

Increment line coverage 16.06% (169/1052) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.85% (19305/36531)
Line Coverage	36.16% (180307/498666)
Region Coverage	32.69% (139424/426491)
Branch Coverage	33.72% (60733/180100)

hello-stephen · 2026-03-17T09:00:49Z

BE Regression && UT Coverage Report

Increment line coverage 16.16% (170/1052) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.34% (25514/35766)
Line Coverage	54.02% (268894/497782)
Region Coverage	51.40% (221421/430766)
Branch Coverage	52.95% (95713/180745)

morningman · 2026-03-17T18:11:30Z

Picked here: #61446

xylaaaaa requested a review from yiguolei as a code owner March 16, 2026 03:10

Copilot AI review requested due to automatic review settings March 16, 2026 03:10

Copilot started reviewing on behalf of xylaaaaa March 16, 2026 03:11 View session

Copilot AI reviewed Mar 16, 2026

View reviewed changes

yiguolei force-pushed the branch-4.1 branch from c123b0d to 0740c34 Compare March 16, 2026 04:45

xylaaaaa force-pushed the auto-pick-58972-branch-4.1 branch from 96798c6 to 92ba67f Compare March 16, 2026 05:12

xylaaaaa and others added 3 commits March 16, 2026 09:59

trigger CI re-run

3740243

morningman force-pushed the auto-pick-58972-branch-4.1 branch from 6a7770f to 7760f61 Compare March 16, 2026 17:00

[fix] fix parquet metadata TVF compile on branch-4.1

8b3fa91

morningman closed this Mar 17, 2026

Conversation

xylaaaaa commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Cherry-pick commit

Uh oh!

Thearas commented Mar 16, 2026

Uh oh!

xylaaaaa commented Mar 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

hello-stephen commented Mar 16, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Mar 16, 2026

FE UT Coverage Report

Uh oh!

xylaaaaa commented Mar 16, 2026

Uh oh!

doris-robot commented Mar 16, 2026

Cloud UT Coverage Report

Uh oh!

yiguolei commented Mar 16, 2026

Uh oh!

hello-stephen commented Mar 16, 2026

FE UT Coverage Report

Uh oh!

morningman commented Mar 16, 2026

Uh oh!

hello-stephen commented Mar 16, 2026

Cloud UT Coverage Report

Uh oh!

xylaaaaa commented Mar 17, 2026

Uh oh!

doris-robot commented Mar 17, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Mar 17, 2026

BE Regression && UT Coverage Report

Uh oh!

morningman commented Mar 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

xylaaaaa commented Mar 16, 2026 •

edited

Loading