Skip to content

Conversation

@zclllyybb
Copy link
Contributor

@zclllyybb zclllyybb commented Jan 30, 2026

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

we have COW and shallow_mutate, so for most of scenarios we dont need to explicitly copy block's data. this PR fixed this for MultiCastOperator. and fixed all founded misuses. added tests.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 30, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zclllyybb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31834 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5193e292eaf17c49e7b6457e53646edcd7fe8e2a, data reload: false

------ Round 1 ----------------------------------
q1	17616	5297	5074	5074
q2	2020	306	184	184
q3	10258	1354	764	764
q4	10192	834	318	318
q5	7473	2140	1880	1880
q6	193	179	153	153
q7	873	747	601	601
q8	9257	1372	1087	1087
q9	5255	4800	4820	4800
q10	6814	1948	1568	1568
q11	488	291	281	281
q12	333	375	229	229
q13	17771	3999	3222	3222
q14	238	235	222	222
q15	895	830	809	809
q16	683	692	635	635
q17	635	813	459	459
q18	6835	6576	6481	6481
q19	1227	979	597	597
q20	379	330	224	224
q21	2585	2039	1971	1971
q22	354	303	275	275
Total cold run time: 102374 ms
Total hot run time: 31834 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5363	5318	5291	5291
q2	257	341	255	255
q3	2148	2687	2229	2229
q4	1367	1748	1331	1331
q5	4304	4209	4180	4180
q6	220	179	140	140
q7	2088	2107	1946	1946
q8	2550	2418	2408	2408
q9	7603	7562	7524	7524
q10	2851	3081	2600	2600
q11	546	516	454	454
q12	680	721	624	624
q13	3817	4448	3547	3547
q14	308	313	296	296
q15	871	816	822	816
q16	682	775	724	724
q17	1141	1428	1388	1388
q18	8216	7878	7680	7680
q19	859	825	819	819
q20	2125	2090	1955	1955
q21	4531	4314	4142	4142
q22	583	537	495	495
Total cold run time: 53110 ms
Total hot run time: 50844 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190155 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5193e292eaf17c49e7b6457e53646edcd7fe8e2a, data reload: false

query5	4391	613	505	505
query6	321	229	220	220
query7	4220	461	254	254
query8	340	256	238	238
query9	8713	2765	2745	2745
query10	449	346	289	289
query11	16384	16456	16110	16110
query12	176	119	119	119
query13	1275	442	355	355
query14	6269	3184	2913	2913
query14_1	2859	2740	2695	2695
query15	194	184	169	169
query16	1000	457	446	446
query17	1059	661	562	562
query18	2434	422	313	313
query19	186	172	145	145
query20	123	119	119	119
query21	207	136	117	117
query22	4940	5142	4729	4729
query23	19074	18725	18273	18273
query23_1	18426	18487	18479	18479
query24	7156	1591	1193	1193
query24_1	1168	1187	1209	1187
query25	510	432	404	404
query26	1233	259	148	148
query27	2773	458	267	267
query28	4540	1880	1877	1877
query29	781	532	453	453
query30	318	249	221	221
query31	877	737	667	667
query32	84	79	79	79
query33	540	378	270	270
query34	877	887	547	547
query35	628	676	597	597
query36	1193	1197	1063	1063
query37	128	98	87	87
query38	2946	2882	2855	2855
query39	931	899	911	899
query39_1	886	875	890	875
query40	215	127	112	112
query41	64	59	61	59
query42	90	91	86	86
query43	424	405	403	403
query44	1277	709	706	706
query45	191	187	179	179
query46	833	953	565	565
query47	2074	2143	2073	2073
query48	307	315	225	225
query49	601	435	336	336
query50	703	262	203	203
query51	4090	4068	4110	4068
query52	90	87	77	77
query53	199	216	171	171
query54	280	247	241	241
query55	80	75	71	71
query56	285	279	289	279
query57	1376	1355	1287	1287
query58	263	259	253	253
query59	2123	2201	2094	2094
query60	322	325	307	307
query61	140	142	141	141
query62	593	555	514	514
query63	199	164	173	164
query64	4871	1142	906	906
query65	4346	4266	4368	4266
query66	1468	455	350	350
query67	17504	17598	17447	17447
query68	2377	1000	651	651
query69	403	313	289	289
query70	1035	914	904	904
query71	307	298	271	271
query72	5443	2608	3303	2608
query73	595	725	283	283
query74	9712	9627	9511	9511
query75	2433	2461	2030	2030
query76	2264	1044	640	640
query77	340	384	293	293
query78	11282	11499	10845	10845
query79	2336	880	558	558
query80	1701	491	453	453
query81	555	280	257	257
query82	986	151	116	116
query83	315	254	242	242
query84	255	121	99	99
query85	857	476	402	402
query86	442	306	295	295
query87	3147	3085	2990	2990
query88	3414	2556	2537	2537
query89	387	332	300	300
query90	1853	173	165	165
query91	163	157	125	125
query92	74	74	64	64
query93	1155	951	570	570
query94	675	274	288	274
query95	594	328	376	328
query96	628	507	224	224
query97	2490	2494	2442	2442
query98	223	221	222	221
query99	982	959	835	835
Total cold run time: 266016 ms
Total hot run time: 190155 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.23 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5193e292eaf17c49e7b6457e53646edcd7fe8e2a, data reload: false

query1	0.05	0.04	0.04
query2	0.10	0.05	0.04
query3	0.25	0.08	0.08
query4	1.60	0.12	0.11
query5	0.27	0.24	0.24
query6	1.19	0.67	0.67
query7	0.03	0.03	0.03
query8	0.05	0.04	0.04
query9	0.56	0.50	0.49
query10	0.55	0.54	0.52
query11	0.14	0.09	0.09
query12	0.13	0.10	0.10
query13	0.62	0.61	0.62
query14	1.06	1.05	1.06
query15	0.87	0.85	0.87
query16	0.39	0.38	0.39
query17	1.15	1.12	1.12
query18	0.22	0.21	0.20
query19	2.02	1.94	2.00
query20	0.02	0.01	0.01
query21	15.42	0.28	0.16
query22	4.93	0.06	0.05
query23	15.89	0.29	0.10
query24	2.42	0.51	0.33
query25	0.11	0.06	0.09
query26	0.15	0.13	0.14
query27	0.08	0.08	0.04
query28	4.64	1.14	0.96
query29	12.56	3.91	3.14
query30	0.28	0.14	0.11
query31	2.80	0.62	0.41
query32	3.24	0.59	0.49
query33	3.34	3.27	3.18
query34	16.26	5.46	4.76
query35	4.87	4.81	4.82
query36	0.65	0.49	0.49
query37	0.11	0.07	0.07
query38	0.08	0.04	0.04
query39	0.05	0.04	0.04
query40	0.18	0.16	0.16
query41	0.10	0.04	0.03
query42	0.04	0.03	0.02
query43	0.06	0.04	0.04
Total cold run time: 99.53 s
Total hot run time: 28.23 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 73.68% (14/19) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.49% (19272/36716)
Line Coverage 35.97% (179072/497791)
Region Coverage 32.39% (138910/428836)
Branch Coverage 33.33% (60092/180275)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 94.74% (18/19) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.58% (25761/35991)
Line Coverage 54.21% (269237/496614)
Region Coverage 51.81% (224472/433246)
Branch Coverage 53.23% (96342/181007)

@zclllyybb zclllyybb marked this pull request as ready for review January 30, 2026 16:33
@zclllyybb
Copy link
Contributor Author

run buildall

Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR removes a redundant deep-copy in the MultiCast data path by relying on COW semantics, and updates several vectorized block/expr mutation sites to avoid in-place mutation when columns are shared. It also adds a regression suite to exercise complex MultiCast/CTE scenarios.

Changes:

  • Remove per-pull column deep copy in MultiCastDataStreamer (switch to shared blocks + COW).
  • Update multiple “clear/filter” paths to respect COW via is_exclusive() checks and clone_empty() fallback.
  • Add a new regression test suite and expected output covering complex CTE MultiCast patterns.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
regression-test/suites/nereids_p0/cte/test_cte_multicast_complex.groovy Adds a regression suite with multiple CTE consumers/filters to trigger MultiCast behavior.
regression-test/data/nereids_p0/cte/test_cte_multicast_complex.out Expected results for the new regression suite.
be/src/vec/exprs/vexpr_context.cpp Makes “filter-all” clearing COW-safe by avoiding in-place clears on shared columns.
be/src/vec/core/block.cpp Makes filter_block_internal/const-filter emptying COW-safe; adds clarifying comments.
be/src/pipeline/exec/nested_loop_join_probe_operator.h Makes block clearing COW-safe in the non-materialize path.
be/src/pipeline/exec/multi_cast_data_streamer.h / .cpp Removes the deep-copy helper and finishes pulls without cloning columns.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@doris-robot
Copy link

TPC-H: Total hot run time: 32031 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f8c3a6050c0ec5e1ad9088dd6b836c99bdef64eb, data reload: false

------ Round 1 ----------------------------------
q1	17630	5300	5061	5061
q2	2045	329	229	229
q3	10164	1307	757	757
q4	10216	894	325	325
q5	7558	2169	1946	1946
q6	197	179	148	148
q7	876	733	611	611
q8	9255	1439	1108	1108
q9	5385	4821	4945	4821
q10	6847	1975	1563	1563
q11	512	287	277	277
q12	374	371	223	223
q13	17781	4079	3244	3244
q14	235	234	223	223
q15	908	826	827	826
q16	683	676	636	636
q17	657	777	532	532
q18	6941	6587	6507	6507
q19	1093	988	609	609
q20	411	350	236	236
q21	2639	2141	1868	1868
q22	366	317	281	281
Total cold run time: 102773 ms
Total hot run time: 32031 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5310	5308	5282	5282
q2	274	329	257	257
q3	2186	2676	2256	2256
q4	1386	1739	1319	1319
q5	4322	4249	4237	4237
q6	219	183	141	141
q7	2390	2079	1936	1936
q8	2608	2404	2423	2404
q9	7486	7386	7539	7386
q10	2868	3071	2634	2634
q11	562	467	464	464
q12	689	778	618	618
q13	4039	4364	3562	3562
q14	285	314	288	288
q15	897	864	812	812
q16	661	727	688	688
q17	1135	1542	1317	1317
q18	7960	8260	8041	8041
q19	949	905	835	835
q20	2060	2193	2062	2062
q21	4845	4390	4479	4390
q22	618	590	510	510
Total cold run time: 53749 ms
Total hot run time: 51439 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.21 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f8c3a6050c0ec5e1ad9088dd6b836c99bdef64eb, data reload: false

query1	0.06	0.05	0.04
query2	0.10	0.04	0.04
query3	0.26	0.09	0.08
query4	1.61	0.12	0.12
query5	0.26	0.27	0.24
query6	1.17	0.68	0.68
query7	0.04	0.03	0.02
query8	0.05	0.03	0.04
query9	0.56	0.51	0.49
query10	0.55	0.54	0.56
query11	0.15	0.09	0.09
query12	0.14	0.10	0.11
query13	0.64	0.61	0.62
query14	1.06	1.05	1.05
query15	0.88	0.88	0.85
query16	0.39	0.40	0.40
query17	1.11	1.15	1.15
query18	0.22	0.20	0.21
query19	2.04	2.01	2.07
query20	0.02	0.01	0.02
query21	15.41	0.26	0.14
query22	5.17	0.06	0.04
query23	15.87	0.29	0.11
query24	1.43	0.69	0.20
query25	0.12	0.09	0.15
query26	0.14	0.15	0.13
query27	0.08	0.07	0.05
query28	4.74	1.16	0.97
query29	12.57	3.99	3.15
query30	0.28	0.13	0.11
query31	2.81	0.65	0.40
query32	3.23	0.59	0.50
query33	3.20	3.19	3.28
query34	16.30	5.40	4.74
query35	4.82	4.80	4.76
query36	0.64	0.50	0.50
query37	0.12	0.07	0.07
query38	0.07	0.04	0.04
query39	0.05	0.03	0.04
query40	0.19	0.15	0.14
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 98.73 s
Total hot run time: 28.21 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 54.55% (18/33) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.48% (19274/36723)
Line Coverage 35.96% (179072/497974)
Region Coverage 32.40% (139010/429052)
Branch Coverage 33.31% (60082/180359)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 90.91% (30/33) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.52% (25746/35998)
Line Coverage 54.16% (269061/496797)
Region Coverage 51.78% (224431/433462)
Branch Coverage 53.15% (96252/181091)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants