Skip to content

[fix](storage) check partition storage policy resource before BE update#65053

Open
gavinchou wants to merge 1 commit into
apache:masterfrom
gavinchou:gavin-cir-19357-storage-policy
Open

[fix](storage) check partition storage policy resource before BE update#65053
gavinchou wants to merge 1 commit into
apache:masterfrom
gavinchou:gavin-cir-19357-storage-policy

Conversation

@gavinchou

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

When ALTER TABLE ... MODIFY PARTITION ... SET ("storage_policy" = ...) changes an existing partition policy to another policy backed by a different storage resource, Doris should reject the operation.

Before this patch, the BE tablet meta update was sent before FE checked the old/new policy resource compatibility. The SQL could fail later in FE, while BE tablet meta had already persisted the new storage_policy_id.

Check List

  • Add a pre-check before updating BE tablet meta.
  • Keep same-resource policy switching allowed.
  • Add regression coverage for MODIFY PARTITION (*) cross-resource failure.

Tests

  • git diff --check

@gavinchou gavinchou requested a review from luwei16 as a code owner June 30, 2026 19:11
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@gavinchou

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29187 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d1244187e57b2f6c65de21a2d7cca21082f24cfd, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17662	3897	3900	3897
q2	1995	312	201	201
q3	10309	1384	817	817
q4	4676	467	335	335
q5	7506	854	579	579
q6	188	166	135	135
q7	768	848	620	620
q8	9698	1668	1566	1566
q9	6261	4421	4416	4416
q10	6840	1793	1518	1518
q11	500	342	309	309
q12	731	560	432	432
q13	18101	3325	2704	2704
q14	294	263	242	242
q15	q16	783	777	699	699
q17	1228	989	885	885
q18	6802	5616	5493	5493
q19	1369	1244	1108	1108
q20	749	675	555	555
q21	6013	2710	2385	2385
q22	424	347	291	291
Total cold run time: 102897 ms
Total hot run time: 29187 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4311	4227	4196	4196
q2	277	311	214	214
q3	4595	4956	4448	4448
q4	2065	2099	1417	1417
q5	4393	4301	4296	4296
q6	230	174	126	126
q7	1776	2004	1720	1720
q8	2419	2174	2056	2056
q9	7833	7759	7711	7711
q10	4770	4758	4279	4279
q11	584	414	395	395
q12	751	742	530	530
q13	3440	3532	2930	2930
q14	321	312	290	290
q15	q16	695	723	686	686
q17	1399	1319	1317	1317
q18	7959	7181	6976	6976
q19	1101	1088	1116	1088
q20	2212	2192	1920	1920
q21	5251	4563	4452	4452
q22	533	450	403	403
Total cold run time: 56915 ms
Total hot run time: 51450 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 173768 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d1244187e57b2f6c65de21a2d7cca21082f24cfd, data reload: false

query5	4326	628	489	489
query6	453	218	224	218
query7	4839	607	343	343
query8	337	185	167	167
query9	8774	4048	4026	4026
query10	489	348	300	300
query11	5957	2327	2083	2083
query12	152	102	99	99
query13	1267	635	463	463
query14	6314	5322	4967	4967
query14_1	4291	4304	4278	4278
query15	214	204	176	176
query16	1007	463	444	444
query17	1111	703	567	567
query18	2470	474	338	338
query19	198	193	156	156
query20	111	105	103	103
query21	232	157	132	132
query22	13753	13642	13404	13404
query23	17430	16497	16042	16042
query23_1	16155	16276	16210	16210
query24	7637	1792	1289	1289
query24_1	1352	1319	1306	1306
query25	553	450	394	394
query26	1395	362	214	214
query27	2545	613	390	390
query28	4482	2051	2044	2044
query29	1121	628	512	512
query30	342	265	241	241
query31	1122	1105	995	995
query32	121	60	61	60
query33	551	326	261	261
query34	1163	1179	662	662
query35	772	788	669	669
query36	1429	1412	1244	1244
query37	155	113	96	96
query38	1901	1725	1665	1665
query39	928	915	886	886
query39_1	883	903	886	886
query40	247	164	145	145
query41	73	69	87	69
query42	105	103	96	96
query43	332	329	287	287
query44	1448	788	777	777
query45	218	191	180	180
query46	1086	1242	762	762
query47	2412	2314	2285	2285
query48	402	432	313	313
query49	588	428	320	320
query50	1141	435	339	339
query51	4475	4423	4270	4270
query52	87	88	76	76
query53	270	292	199	199
query54	296	242	254	242
query55	77	72	69	69
query56	306	317	319	317
query57	1437	1414	1350	1350
query58	275	271	248	248
query59	1571	1655	1429	1429
query60	316	282	250	250
query61	179	175	177	175
query62	707	651	590	590
query63	256	209	219	209
query64	2623	755	601	601
query65	4855	4788	4779	4779
query66	1861	516	390	390
query67	29776	29657	29447	29447
query68	3101	1564	1006	1006
query69	414	297	274	274
query70	1044	982	924	924
query71	348	319	299	299
query72	2810	2588	2318	2318
query73	816	763	444	444
query74	5073	5007	4733	4733
query75	2600	2600	2248	2248
query76	2305	1162	807	807
query77	350	387	293	293
query78	12304	12532	11857	11857
query79	1399	1142	762	762
query80	929	538	481	481
query81	490	333	275	275
query82	571	160	123	123
query83	368	334	294	294
query84	282	162	130	130
query85	936	611	525	525
query86	401	310	272	272
query87	1839	1835	1769	1769
query88	3754	2839	2791	2791
query89	466	409	357	357
query90	1879	204	197	197
query91	198	193	160	160
query92	63	63	53	53
query93	1552	1507	912	912
query94	595	366	313	313
query95	785	609	474	474
query96	1109	863	330	330
query97	2705	2709	2577	2577
query98	214	205	199	199
query99	1189	1150	1029	1029
Total cold run time: 259064 ms
Total hot run time: 173768 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.17 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d1244187e57b2f6c65de21a2d7cca21082f24cfd, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.06	0.05
query3	0.26	0.13	0.13
query4	1.60	0.14	0.13
query5	0.23	0.23	0.22
query6	1.27	1.05	1.05
query7	0.03	0.00	0.00
query8	0.06	0.03	0.04
query9	0.37	0.31	0.30
query10	0.59	0.53	0.59
query11	0.18	0.14	0.14
query12	0.18	0.14	0.14
query13	0.46	0.46	0.47
query14	1.01	1.01	1.01
query15	0.60	0.59	0.58
query16	0.32	0.31	0.31
query17	1.14	1.11	1.12
query18	0.23	0.21	0.20
query19	2.04	1.98	1.94
query20	0.02	0.02	0.01
query21	15.43	0.19	0.14
query22	4.93	0.06	0.06
query23	16.14	0.30	0.12
query24	2.96	0.41	0.32
query25	0.11	0.05	0.04
query26	0.73	0.21	0.16
query27	0.03	0.04	0.03
query28	3.62	0.91	0.56
query29	12.50	4.26	3.50
query30	0.28	0.16	0.16
query31	2.78	0.59	0.30
query32	3.22	0.59	0.49
query33	3.18	3.35	3.16
query34	15.51	4.17	3.48
query35	3.50	3.52	3.53
query36	0.55	0.44	0.43
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.03
query40	0.17	0.16	0.15
query41	0.08	0.02	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 96.67 s
Total hot run time: 25.17 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants